Deep Learning for Multi-path Error Removal in ToF Sensors

  • Gianluca Agresti
  • Pietro ZanuttighEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11131)


The removal of Multi-Path Interference (MPI) is one of the major open challenges in depth estimation with Time-of-Flight (ToF) cameras. In this paper we propose a novel method for MPI removal and depth refinement exploiting an ad-hoc deep learning architecture working on data from a multi-frequency ToF camera. In order to estimate the MPI we use a Convolutional Neural Network (CNN) made of two sub-networks: a coarse network analyzing the global structure of the data at a lower resolution and a fine one exploiting the output of the coarse network in order to remove the MPI while preserving the small details. The critical issue of the lack of ToF data with ground truth is solved by training the CNN with synthetic information. Finally, the residual zero-mean error is removed with an adaptive bilateral filter guided from a noise model for the camera. Experimental results prove the effectiveness of the proposed approach on both synthetic and real data.


ToF sensors Denoising Multi-path interference Depth acquisition Convolutional Neural Networks 



We would like to thank the Computational Imaging Group at the Sony European Technology Center (EuTEC) for allowing us to use their ToF Explorer simulator and Oliver Erdler, Markus Kamm and Henrik Schaefer for their precious comments and insights. We also thank prof. Calvagno for his support and gratefully acknowledge NVIDIA Corporation for the donation of the GPUs used for this research.

Supplementary material

478822_1_En_30_MOESM1_ESM.pdf (1.1 mb)
Supplementary material 1 (pdf 1118 KB)


  1. 1.
    Zanuttigh, P., Marin, G., Dal Mutto, C., Dominio, F., Minto, L., Cortelazzo, G.M.: Time-of-Flight and Structured Light Depth Cameras. Springer, Switzerland (2016). Scholar
  2. 2.
    Freedman, D., Smolin, Y., Krupka, E., Leichter, I., Schmidt, M.: SRA: fast removal of general multipath for ToF sensors. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 234–249. Springer, Cham (2014). Scholar
  3. 3.
    Bhandari, A., et al.: Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Optics Lett. 39(6), 1705–1708 (2014)CrossRefGoogle Scholar
  4. 4.
    Peters, C., Klein, J., Hullin, M.B., Klein, R.: Solving trigonometric moment problems for fast transient imaging. ACM Trans. Graph. (TOG) 34(6), 220 (2015)CrossRefGoogle Scholar
  5. 5.
    Son, K., Liu, M.Y., Taguchi, Y.: Learning to remove multipath distortions in time-of-flight range images for a robotic arm setup. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 3390–3397 (2016)Google Scholar
  6. 6.
    Marco, J., et al.: Deeptof: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (TOG) 36(6), 219 (2017)CrossRefGoogle Scholar
  7. 7.
    Agresti, G., Minto, L., Marin, G., Zanuttigh, P.: Deep learning for confidence information in stereo and ToF data fusion. In: Geometry Meets Deep Learning ICCV Workshop, pp. 697–705 (2017)Google Scholar
  8. 8.
    Su, S., Heide, F., Wetzstein, G., Heidrich, W.: Deep end-to-end time-of-flight imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6383–6392 (2018)Google Scholar
  9. 9.
    Guo, Q., Frosio, I., Gallo, O., Zickler, T., Kautz, J.: Tackling 3D ToF artifacts through learning and the FLAT dataset. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 381–396. Springer, Cham (2018). Scholar
  10. 10.
    Kadambi, A., et al.: Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. (TOG) 32(6), 167 (2013)CrossRefGoogle Scholar
  11. 11.
    Gupta, M., Nayar, S.K., Hullin, M.B., Martin, J.: Phasor imaging: a generalization of correlation-based time-of-flight imaging. ACM Trans. Graph. (TOG) 34(5), 156 (2015)CrossRefGoogle Scholar
  12. 12.
    Whyte, R., Streeter, L., Cree, M.J., Dorrington, A.A.: Review of methods for resolving multi-path interference in time-of-flight range cameras. In: IEEE Sensors, pp. 629–632. IEEE (2014)Google Scholar
  13. 13.
    Fuchs, S.: Multipath interference compensation in time-of-flight camera images. In: 2010 20th International Conference on Pattern Recognition (ICPR), pp. 3583–3586. IEEE (2010)Google Scholar
  14. 14.
    Fuchs, S., Suppa, M., Hellwich, O.: Compensation for multipath in ToF camera measurements supported by photometric calibration and environment integration. In: Chen, M., Leibe, B., Neumann, B. (eds.) ICVS 2013. LNCS, vol. 7963, pp. 31–41. Springer, Heidelberg (2013). Scholar
  15. 15.
    Jiménez, D., Pizarro, D., Mazo, M., Palazuelos, S.: Modeling and correction of multipath interference in time of flight cameras. Image Vis. Comput. 32(1), 1–13 (2014)CrossRefGoogle Scholar
  16. 16.
    Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., Bing Kang, S.: A light transport model for mitigating multipath interference in time-of-flight sensors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 73–81 (2015)Google Scholar
  17. 17.
    Whyte, R., Streeter, L., Cree, M.J., Dorrington, A.A.: Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods. Opt. Eng. 54(11), 113109 (2015)CrossRefGoogle Scholar
  18. 18.
    Agresti, G., Zanuttigh, P.: Combination of spatially-modulated ToF and structured light for MPI-free depth estimation. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018 Workshops. LNCS, vol. 11129, pp. 355–371. Springer, Cham (2018)Google Scholar
  19. 19.
    Nayar, S.K., Krishnan, G., Grossberg, M.D., Raskar, R.: Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. Graph. (TOG) 25(3), 935–944 (2006)CrossRefGoogle Scholar
  20. 20.
    Luo, W., Schwing, A.G., Urtasun, R.: Efficient deep learning for stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5695–5703 (2016)Google Scholar
  21. 21.
    Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: Advances in Neural Information Processing Systems, pp. 2366–2374 (2014)Google Scholar
  22. 22.
    Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2650–2658 (2015)Google Scholar
  23. 23.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of International Conference on Computer Vision (ICCV), pp. 839–846. IEEE (1998)Google Scholar
  24. 24.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A noise-aware filter for real-time depth upsampling. In: Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications-M2SFA2 2008 (2008)Google Scholar
  25. 25.
    Marin, G., Zanuttigh, P., Mattoccia, S.: Reliable fusion of ToF and stereo depth driven by confidence measures. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 386–401. Springer, Cham (2016). Scholar
  26. 26.
    Zhang, M., Gunturk, B.K.: Multiresolution bilateral filtering for image denoising. IEEE Trans. Image Process. 17(12), 2324–2333 (2008)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Lange, R., Seitz, P., Biber, A., Lauxtermann, S.C.: Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging. In: Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications, vol. 3965, pp. 177–189. International Society for Optics and Photonics (2000)Google Scholar
  28. 28.
    Meister, S., Nair, R., Kondermann, D.: Simulation of time-of-flight sensors using global illumination. In: Bronstein, M., Favre, J., Hormann, K. (eds.) Vision, Modeling and Visualization. The Eurographics Association, Goslar (2013)Google Scholar
  29. 29.
    The Blender Foundation: Blender website. Accessed 14 Mar 2018
  30. 30.
    The LuxRender Project: Luxrender website. Accessed 14 Mar 2018
  31. 31.
    Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2017)CrossRefGoogle Scholar
  32. 32.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  33. 33.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversity of PadovaPadovaItaly

Personalised recommendations