A Survey on Time-of-Flight Stereo Fusion

  • Rahul Nair
  • Kai Ruhl
  • Frank Lenzen
  • Stephan Meister
  • Henrik Schäfer
  • Christoph S. Garbe
  • Martin Eisemann
  • Marcus Magnor
  • Daniel Kondermann
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8200)


Due to the demand for depth maps of higher quality than possible with a single depth imaging technique today, there has been an increasing interest in the combination of different depth sensors to produce a “super-camera” that is more than the sum of the individual parts. In this survey paper, we give an overview over methods for the fusion of Time-of-Flight (ToF) and passive stereo data as well as applications of the resulting high quality depth maps. Additionally, we provide a tutorial-based introduction to the principles behind ToF stereo fusion and the evaluation criteria used to benchmark these methods.


Fusion System Stereo Match Structure From Motion Camera Setup Stereo Data 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Park, J., Kim, H., Tai, Y., Brown, M., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: IEEE Proc. ICCV (2011)Google Scholar
  2. 2.
    Huhle, B., Fleck, S., Schilling, A.: Integrating 3d time-of-flight camera data and high resolution images for 3dtv applications. In: Proc. 3DTV Conf. IEEE (2007)Google Scholar
  3. 3.
    Castaneda, V., Mateus, D., Navab, N.: Stereo time-of-flight. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1684–1691. IEEE (2011)Google Scholar
  4. 4.
    Kuhnert, K., Stommel, M.: Fusion of stereo-camera and pmd-camera data for real-time suited precise 3d environment reconstruction. In: Int. Conf. on Intelligent Robots and Systems, pp. 4780–4785. IEEE (2006)Google Scholar
  5. 5.
    Fischer, J., Arbeiter, G., Verl, A.: Combination of time-of-flight depth and stereo using semiglobal optimization. In: Int. Conf. on Robotics and Automation (ICRA), pp. 3548–3553. IEEE (2011)Google Scholar
  6. 6.
    Eisemann, E., Durand, F.: Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics (Proc. of SIGGRAPH) 23 (2004)Google Scholar
  7. 7.
    Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23(3), 664–672 (2004)CrossRefGoogle Scholar
  8. 8.
    Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., Durand, F.: The video mesh: A data structure for image-based three-dimensional video editing. In: Proc. of the International Conference on Computional Photography (ICCP) (2011)Google Scholar
  9. 9.
    Lo, W.Y., van Baar, J., Knaus, C., Zwicker, M., Gross, M.: Stereoscopic 3d copy & paste. ACM Trans. Graph. 29(6), 147:1–147:10 (2010)Google Scholar
  10. 10.
    Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S.A.J., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23(3), 600–608 (2004)CrossRefGoogle Scholar
  11. 11.
    Devernay, F., Beardsley, P.: Stereoscopic Cinema. In: Ronfard, R., Taubin, G. (eds.) Image and Geometry Processing for 3-D Cinematography. Geometry and Computing, vol. 5, pp. 11–51. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  12. 12.
    Fei, Y., Yu, M., Shao, F., Jiang, G.: A color correction algorithm of multi-view video based on depth segmentation. In: International Symposium on Computer Science and Computational Technology (ISCSCT 2008), vol. 2, pp. 206–209 (2008)Google Scholar
  13. 13.
    Wilkes, L.: The role of ocula in stereo post production. The Foundry. Whitepaper (2009)Google Scholar
  14. 14.
    Templin, K., Didyk, P., Ritschel, T., Myszkowski, K., Seidel, H.P.: Highlight microdisparity for improved gloss depiction. ACM Transactions on Graphics (Proc. SIGGRAPH) 31(4) (2012)Google Scholar
  15. 15.
    Mcmillan, L., Gortler, S.: Image-based rendering: A new interface between computer vision and computer graphics. SIGGRAPH Comput. Graph. 33, 61–64 (2000)CrossRefGoogle Scholar
  16. 16.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1), 7–42 (2002)CrossRefzbMATHGoogle Scholar
  17. 17.
    Chiu, W.C., Blanke, U., Fritz, M.: Improving the kinect by cross-modal stereo. In: British Machine Vision Conf. BMVA, pp. 116–1 (2011)Google Scholar
  18. 18.
    Hrkać, T., Kalafatić, Z., Krapac, J.: Infrared-visual image registration based on corners and hausdorff distance. In: Ersbøll, B.K., Pedersen, K.S. (eds.) SCIA 2007. LNCS, vol. 4522, pp. 383–392. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  19. 19.
    Bilodeau, G., Torabi, A., Morin, F.: Visible and infrared image registration using trajectories and composite foreground images. Image and Vision Computing 29(1), 41–50 (2011)CrossRefGoogle Scholar
  20. 20.
    Toet, A., Van Ruyven, L.J., Valeton, J.M.: Merging thermal and visual images by a contrast pyramid. Optical Engineering 28(7), 287789–287789 (1989)CrossRefGoogle Scholar
  21. 21.
    Wu, C.: Visualsfm: A visual structure from motion system (2011)Google Scholar
  22. 22.
    Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: Dtam: Dense tracking and mapping in real-time. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2320–2327. IEEE (2011)Google Scholar
  23. 23.
    Horn, B.K., Brooks, M.J.: Shape from shading. MIT Press (1989)Google Scholar
  24. 24.
    Barron, J.T., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 334–341. IEEE (2012)Google Scholar
  25. 25.
    Sturmer, M., Penne, J., Hornegger, J.: Standardization of intensity-values acquired by time-of-flight-cameras. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2008), pp. 1–6. IEEE (2008)Google Scholar
  26. 26.
    Kim, Y., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., Thrun, S.: Multi-view image and tof sensor fusion for dense 3d reconstruction. In: ICCV Workshops, pp. 1542–1549. IEEE (2009)Google Scholar
  27. 27.
    Zhu, J., Wang, L., Gao, J., Yang, R.: Spatial-temporal fusion for high accuracy depth maps using dynamic mrfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(5), 899–909 (2010)CrossRefGoogle Scholar
  28. 28.
    Ghobadi, S.E., Loepprich, O.E., Lottnera, O., Ahmadov, F., Hartmann, K., Weihs, W., Loffeld, O.: Analysis of the personnel safety in a man-machine-cooperation using 2d/3d images. In: Proceedings of the EURON/IARP International Workshop on Robotics for Risky Interventions and Surveillance of the Environment, Benicassim, Spain (January 2008)Google Scholar
  29. 29.
    Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)Google Scholar
  30. 30.
    Bouguet, J.Y.: Camera calibration toolbox for matlab (2004)Google Scholar
  31. 31.
    Hahne, U., Alexa, M.: Combining time-of-flight depth and stereo images without accurate extrinsic calibration. IJISTA 5(3), 325–333 (2008)CrossRefGoogle Scholar
  32. 32.
    Nair, R., Lenzen, F., Meister, S., Schäfer, H., Garbe, C., Kondermann, D.: High accuracy TOF and stereo sensor fusion at interactive rates. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part II. LNCS, vol. 7584, pp. 1–11. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  33. 33.
    Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: A probabilistic approach to tof and stereo data fusion. In: 3DPVT, Paris, France, vol. 2 (2010)Google Scholar
  34. 34.
    Schiller, I., Beder, C., Koch, R.: Calibration of a pmd-camera using a planar calibration pattern together with a multi-camera setup. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 37, 297–302 (2008)Google Scholar
  35. 35.
    Zhu, J., Wang, L., Yang, R., Davis, J.: Fusion of time-of-flight depth and stereo for high accuracy depth maps. In: Proc. CVPR, pp. 1–8. IEEE (2008)Google Scholar
  36. 36.
    Kim, Y.M., Chan, D., Theobalt, C., Thrun, S.: Design and calibration of a multi-view tof sensor fusion system. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2008), pp. 1–7. IEEE (2008)Google Scholar
  37. 37.
    Horn, B.K.: Closed-form solution of absolute orientation using unit quaternions. JOSA A 4(4), 629–642 (1987)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Guan, L., Pollefeys, M., et al.: A unified approach to calibrate a network of camcorders and tof cameras. In: Workshop on Multi-Camera and Multi-Modal Sensor Fusion Algorithms and Applications (M2SFA2 2008) (2008)Google Scholar
  39. 39.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press (2004)Google Scholar
  40. 40.
    Dal Mutto, C., Zanuttigh, P., Mattoccia, S., Cortelazzo, G.: Locally consistent toF and stereo data fusion. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part I. LNCS, vol. 7583, pp. 598–607. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  41. 41.
    Zhang, L., Curless, B., Seitz, S.M.: Spacetime stereo: Shape recovery for dynamic scenes. In: Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. II–367. IEEE (2003)Google Scholar
  42. 42.
    Yang, Q., Tan, K.H., Culbertson, B., Apostolopoulos, J.: Fusion of active and passive sensors for fast 3d capture. In: MMSP (2010)Google Scholar
  43. 43.
    Reynolds, M., Dobos, J., Peel, L., Weyrich, T., Brostow, G.J.: Capturing time-of-flight data with confidence. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 945–952. IEEE (2011)Google Scholar
  44. 44.
    Zhu, J., Wang, L., Yang, R., Davis, J., et al.: Reliability fusion of time-of-flight depth and stereo for high quality depth maps. TPAMI (99), 1 (2011)Google Scholar
  45. 45.
    Song, Y., Glasbey, C.A., van der Heijden, G.W.A.M., Polder, G., Dieleman, J.A.: Combining stereo and time-of-flight images with application to automatic plant phenotyping. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 467–478. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  46. 46.
    Gudmundsson, S., Aanaes, H., Larsen, R.: Fusion of stereo vision and time-of-flight imaging for improved 3d estimation. IJISTA 5(3), 425–433 (2008)CrossRefGoogle Scholar
  47. 47.
    Bartczak, B., Koch, R.: Dense depth maps from low resolution time-of-flight depth and high resolution color views. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Kuno, Y., Wang, J., Pajarola, R., Lindstrom, P., Hinkenjann, A., Encarnação, M.L., Silva, C.T., Coming, D. (eds.) ISVC 2009, Part II. LNCS, vol. 5876, pp. 228–239. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  48. 48.
    Hahne, U., Alexa, M.: Depth imaging by combining time-of-flight and on-demand stereo. In: Kolb, A., Koch, R. (eds.) Dyn3D 2009. LNCS, vol. 5742, pp. 70–83. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  49. 49.
    Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: A probabilistic approach to tof and stereo data fusion. In: 3DPVT, Paris, France (May 2010)Google Scholar
  50. 50.
    Beder, C., Bartczak, B., Koch, R.: A combined approach for estimating patchlets from PMD depth images and stereo intensity images. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 11–20. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  51. 51.
    Ruhl, K., Klose, F., Lipski, C., Magnor, M.: Integrating approximate depth data into dense image correspondence estimation. In: Proc. European Conference on Visual Media Production (CVMP) (August 2012)Google Scholar
  52. 52.
    Gandhi, V., Cech, J., Horaud, R.: High-resolution depth maps based on tof-stereo fusion. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 4742–4749. IEEE (2012)Google Scholar
  53. 53.
    Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L.: A hierarchical symmetric stereo algorithm using dynamic programming. International Journal of Computer Vision 47(1-3), 275–285 (2002)CrossRefzbMATHGoogle Scholar
  54. 54.
    Gallup, D., Frahm, J.M., Mordohai, P., Yang, Q., Pollefeys, M.: Real-time plane-sweeping stereo with multiple sweeping directions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2007), pp. 1–8. IEEE (2007)Google Scholar
  55. 55.
    Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusions using graph cuts. In: Proceedings of the Eighth IEEE International Conference on Computer Vision (ICCV 2001), vol. 2, pp. 508–515. IEEE (2001)Google Scholar
  56. 56.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  57. 57.
    Sun, J., Zheng, N.N., Shum, H.Y.: Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(7), 787–800 (2003)CrossRefGoogle Scholar
  58. 58.
    Ihler, A.T., Mcallester, D.A.: Particle belief propagation. In: International Conference on Artificial Intelligence and Statistics, pp. 256–263 (2009)Google Scholar
  59. 59.
    Birchfield, S., Tomasi, C.: A pixel dissimilarity measure that is insensitive to image sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(4), 401–406 (1998)CrossRefGoogle Scholar
  60. 60.
    Hirschmüller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: Proc. CVPR, pp. 1–8. IEEE (2007)Google Scholar
  61. 61.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  62. 62.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. In: Pattern Recognition: 29th DAGM Symposium, vol. 29, pp. 214–223 (2007)Google Scholar
  63. 63.
    Nocedal, J.: Updating quasi-newton matrices with limited storage. Mathematics of Computation 25, 773–782 (1980)MathSciNetCrossRefGoogle Scholar
  64. 64.
    Hirschmüller, H.: Stereo processing by semiglobal matching and mutual information. TPAMI 30(2), 328–341 (2008)CrossRefGoogle Scholar
  65. 65.
    Mattoccia, S.: A locally global approach to stereo correspondence. In: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1763–1770. IEEE (2009)Google Scholar
  66. 66.
    Cech, J., Sara, R.: Efficient sampling of disparity space for fast and accurate matching. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2007), pp. 1–8. IEEE (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Rahul Nair
    • 1
    • 2
  • Kai Ruhl
    • 3
  • Frank Lenzen
    • 1
    • 2
  • Stephan Meister
    • 1
    • 2
  • Henrik Schäfer
    • 1
    • 2
  • Christoph S. Garbe
    • 1
    • 2
  • Martin Eisemann
    • 3
  • Marcus Magnor
    • 3
  • Daniel Kondermann
    • 1
    • 2
  1. 1.Heidelberg Collaboratory for Image Processing (HCI)Heidelberg UniversityGermany
  2. 2.Intel Visual Computing InstituteSaarland UniversityGermany
  3. 3.Technische Universität BraunschweigGermany

Personalised recommendations