High Accuracy TOF and Stereo Sensor Fusion at Interactive Rates

  • Rahul Nair
  • Frank Lenzen
  • Stephan Meister
  • Henrik Schäfer
  • Christoph Garbe
  • Daniel Kondermann
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7584)


We propose two new GPU-based sensor fusion approaches for time of flight (TOF) and stereo depth data. Data fidelity measures are defined to deal with the fundamental limitations of both techniques alone. Our algorithms combine TOF and stereo, yielding megapixel depth maps, enabling our approach to be used in a movie production scenario. Our local model works at interactive rates but yields noisier results, whereas our variational technique is more robust at a higher computational cost. The results show an improvement over each individual method with TOF interreflection remaining an open challenge. To encourage quantitative evaluations, a ground truth dataset is made publicly available.


Interactive Rate Sensor Fusion Stereo Match Total Variation Regularization Ground Truth Dataset 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Xu, Z., Schwarte, R., Heinol, H., Buxbaum, B., Ringbeck, T.: Smart pixel–photonic mixer device (pmd). In: Proc. Int. Conf. on Mechatron. & Machine Vision (1998)Google Scholar
  2. 2.
    Lange, R., Seitz, P.: Solid-state time-of-flight range camera. JQE 37 (2001)Google Scholar
  3. 3.
    Schmidt, M.: Dissertation, IWR, Faculty of Physics, Univ. Heidelberg (2011)Google Scholar
  4. 4.
    Reynolds, M., Dobos, J., Peel, L., Weyrich, T., Brostow, G.: Capturing time-of-flight data with confidence. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 945–952. IEEE (2011)Google Scholar
  5. 5.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47, 7–42 (2002)zbMATHCrossRefGoogle Scholar
  6. 6.
    Huhle, B., Fleck, S., Schilling, A.: Integrating 3d time-of-flight camera data and high resolution images for 3dtv applications. In: Proc. 3DTV Conf. IEEE (2007)Google Scholar
  7. 7.
    Park, J., Kim, H., Tai, Y., Brown, M., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: IEEE Proc. ICCV (2011)Google Scholar
  8. 8.
    Kuhnert, K., Stommel, M.: Fusion of stereo-camera and pmd-camera data for real-time suited precise 3d environment reconstruction. In: Int. Conf. on Intelligent Robots and Systems, pp. 4780–4785. IEEE (2006)Google Scholar
  9. 9.
    Gudmundsson, S., Aanaes, H., Larsen, R.: Fusion of stereo vision and time-of-flight imaging for improved 3d estimation. IJISTA 5, 425–433 (2008)CrossRefGoogle Scholar
  10. 10.
    Bartczak, B., Koch, R.: Dense Depth Maps from Low Resolution Time-of-Flight Depth and High Resolution Color Views. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Kuno, Y., Wang, J., Pajarola, R., Lindstrom, P., Hinkenjann, A., Encarnação, M.L., Silva, C.T., Coming, D. (eds.) ISVC 2009, Part II. LNCS, vol. 5876, pp. 228–239. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  11. 11.
    Hahne, U., Alexa, M.: Depth imaging by combining time-of-flight and on-demand stereo. In: Dynamic 3D Imaging, pp. 70–83 (2009)Google Scholar
  12. 12.
    Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: A probabilistic approach to tof and stereo data fusion. In: 3DPVT, Paris, France (2010)Google Scholar
  13. 13.
    Yang, Q., Tan, K.H., Culbertson, B., Apostolopoulos, J.: Fusion of active and passive sensors for fast 3d capture. In: MMSP (2010)Google Scholar
  14. 14.
    Fischer, J., Arbeiter, G., Verl, A.: Combination of time-of-flight depth and stereo using semiglobal optimization. In: Int. Conf. on Robotics and Automation (ICRA), pp. 3548–3553. IEEE (2011)Google Scholar
  15. 15.
    Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. TPAMI 30, 328–341 (2008)CrossRefGoogle Scholar
  16. 16.
    Hahne, U., Alexa, M.: Combining time-of-flight depth and stereo images without accurate extrinsic calibration. IJISTA 5, 325–333 (2008)CrossRefGoogle Scholar
  17. 17.
    Zhu, J., Wang, L., Yang, R., Davis, J., et al.: Reliability fusion of time-of-flight depth and stereo for high quality depth maps. TPAMI, 1 (2011)Google Scholar
  18. 18.
    Kim, Y., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., Thrun, S.: Multi-view image and tof sensor fusion for dense 3d reconstruction. In: ICCV Workshops, pp. 1542–1549. IEEE (2009)Google Scholar
  19. 19.
    Trobin, W., Pock, T., Cremers, D., Bischof, H.: An Unbiased Second-Order Prior for High-Accuracy Motion Estimation. In: Rigoll, G. (ed.) DAGM 2008. LNCS, vol. 5096, pp. 396–405. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An Improved Algorithm for TV-L 1 Optical Flow. In: Cremers, D., Rosenhahn, B., Yuille, A.L., Schmidt, F.R. (eds.) Visual Motion Analysis. LNCS, vol. 5604, pp. 23–45. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  21. 21.
    Werlberger, M., Unger, M., Pock, T., Bischof, H.: Efficient Minimization of the Non-local Potts Model. In: Bruckstein, A.M., ter Haar Romeny, B.M., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 314–325. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  22. 22.
    Lenzen, F., Schäfer, H., Garbe, C.: Denoising Time-Of-Flight Data with Adaptive Total Variation. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Wang, S., Kyungnam, K., Benes, B., Moreland, K., Borst, C., DiVerdi, S. (eds.) ISVC 2011, Part I. LNCS, vol. 6938, pp. 337–346. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  23. 23.
    Horn, B., Schunck, B.: Determining optical flow. AI 17, 185–204 (1981)Google Scholar
  24. 24.
    Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: Highly accurate optic flow computation with theoretically justified warping. IJCV 67, 141–158 (2006)CrossRefGoogle Scholar
  25. 25.
    Köthe, U.: Edge and Junction Detection with an Improved Structure Tensor. In: Michaelis, B., Krell, G. (eds.) DAGM 2003. LNCS, vol. 2781, pp. 25–32. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  26. 26.
    Canny, J.F.: A Computational Approach to Edge Detection. TPAMI 8 (1986)Google Scholar
  27. 27.
    Setzer, S., Steidl, G., Teuber, T.: Infimal convolution regularizations with discrete l1-type functionals. Comm. Math. Sci. 9, 797–872 (2011)MathSciNetGoogle Scholar
  28. 28.
    Clason, C., Jin, B., Kunisch, K.: A duality-based splitting method for l1-tv image restoration with automatic regularization parameter choice. SIAM J. Sci. Comp., 1484–1505 (2010)Google Scholar
  29. 29.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. JMIV 40, 120–145 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    Hirschmüller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: Proc. CVPR, pp. 1–8. IEEE (2007)Google Scholar
  31. 31.
    Goral, C., Torrance, K., Greenberg, D., Battaile, B.: Modeling the interaction of light between diffuse surfaces. In: ACM SIGGRAPH Computer Graphics, vol. 18, pp. 213–222. ACM (1984)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Rahul Nair
    • 1
    • 2
  • Frank Lenzen
    • 1
    • 2
  • Stephan Meister
    • 1
    • 2
  • Henrik Schäfer
    • 1
    • 2
  • Christoph Garbe
    • 1
    • 2
  • Daniel Kondermann
    • 1
    • 2
  1. 1.Heidelberg Collaboratory for Image ProcessingHeidelberg UniversityGermany
  2. 2.Intel Visual Computing InstituteSaarland UniversityGermany

Personalised recommendations