Denoising Strategies for Time-of-Flight Data

  • Frank Lenzen
  • Kwang In Kim
  • Henrik Schäfer
  • Rahul Nair
  • Stephan Meister
  • Florian Becker
  • Christoph S. Garbe
  • Christian Theobalt
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8200)


When considering the task of denoising ToF data, two issues arise concerning the optimal strategy. The first one is the choice of an appropriate denoising method and its adaptation to ToF data, the second one is the issue of the optimal positioning of the denoising step within the processing pipeline between acquisition of raw data of the sensor and the final output of the depth map. Concerning the first issue, several denoising approaches specifically for ToF data have been proposed in literature, and one contribution of this chapter is to provide an overview. To tackle the second issue, we exemplarily focus on two state-of-the-art methods, the bilateral filtering and total variation (TV) denoising and discuss several alternatives of positions in the pipeline, where these methods can be applied. In our experiments, we compare and evaluate the results of each combination of method and position both qualitatively and quantitatively. It turns out, that for TV denoising the optimal position is at the very end of the pipeline. For the bilateral filter, a quantitative comparison shows that applying it to the raw data together with a subsequent median filtering provides a low error to ground truth. Qualitatively, it competes with applying the (cross-)bilateral filter to the depth data. In particular, the optimal position in general depends on the considered method. As a consequence, for any newly introduced denoising technique, finding its optimal position within the pipeline is an open issue.


Mean Square Error Ground Truth Depth Data Image Denoising Bilateral Filter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aurich, V., Weule, J.: Non-linear Gaussian filters performing edge preserving diffusion. In: Proceed. 17. DAGM-Symposium (1995)Google Scholar
  2. 2.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of the Sixth International Conference on Computer Vision (ICCV 1998), p. 839 (1998)Google Scholar
  3. 3.
    Elad, M.: On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing 11(10), 1141–1151 (2002)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Perona, P., Shiota, T., Malik, J.: Anisotropic diffusion. In: Geometry-Driven Diffusion in Computer Vision, pp. 73–92. Springer (1994)Google Scholar
  5. 5.
    Weickert, J.: Anisotropic diffusion in image processing, vol. 1. Teubner Stuttgart (1998)Google Scholar
  6. 6.
    Donoho, D.L., Johnstone, J.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60(1-4), 259–268 (1992)Google Scholar
  8. 8.
    Grasmair, M.: Locally adaptive total variation regularization. In: Tai, X.-C., Mørken, K., Lysaker, M., Lie, K.-A. (eds.) SSVM 2009. LNCS, vol. 5567, pp. 331–342. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  9. 9.
    Dong, Y.: Multi-scale total variation with automated regularization parameter selection for color image restoration. In: Tai, X.-C., Mørken, K., Lysaker, M., Lie, K.-A. (eds.) SSVM 2009. LNCS, vol. 5567, pp. 271–281. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  10. 10.
    Steidl, G., Teuber, T.: Anisotropic smoothing using double orientations. In: Tai, X.-C., Mørken, K., Lysaker, M., Lie, K.-A. (eds.) SSVM 2009. LNCS, vol. 5567, pp. 477–489. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  11. 11.
    Lenzen, F., Becker, F., Lellmann, J., Petra, S., Schnörr, C.: A class of quasi-variational inequalities for adaptive image denoising and decomposition. Computational Optimization and Applications, 1–28 (2013)Google Scholar
  12. 12.
    Bredies, K., Kunisch, K., Pock, T.: Total Generalized Variation. SIAM J. Imaging Sciences 3(3), 492–526 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Setzer, S., Steidl, G., Teuber, T.: Infimal convolution regularizations with discrete l1-type functionals. Comm. Math. Sci. 9, 797–872 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Lenzen, F., Becker, F., Lellmann, J.: Adaptive second-order total variation: An approach aware of slope discontinuities. In: Pack, T. (ed.) SSVM 2013. LNCS, vol. 7893, pp. 61–73. Springer, Heidelberg (2013)Google Scholar
  15. 15.
    Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Communications on Pure and Applied Mathematics 42(5), 577–685 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the piecewise smooth mumford-shah functional. In: IEEE International Conference on Computer Vision (ICCV), Kyoto, Japan (2009)Google Scholar
  17. 17.
    Buades, A., Coll, B., Morel, J.: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Gilboa, G., Osher, S.: Nonlocal operators with applications to image processing. Multiscale Model. Simul. 7(3), 1005–1028 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Kindermann, S., Osher, S., Jones, P.: Deblurring and denoising of images by nonlocal functionals. Multiscale Model. Simul. 4(4), 1091–1115 (2005) (electronic)Google Scholar
  20. 20.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: Can plain neural networks compete with BM3D? In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2012), pp. 2392–2399. IEEE (2012)Google Scholar
  22. 22.
    Jain, V., Seung, H.S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2008)Google Scholar
  23. 23.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Computer Graphics and Applications 22(2), 56–65 (2002)CrossRefGoogle Scholar
  24. 24.
    Tappen, M.F., Russel, B.C., Freeman, W.T.: Exploiting the sparse derivative prior for super-resolution and image demosaicing. In: Proc. International Workshop on Statistical and Computational Theories of Vision (2003)Google Scholar
  25. 25.
    Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE Trans. Pattern Analysis and Machine Intelligence 32(6), 1127–1133 (2010)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Kim, K.I., Kwon, Y., Kim, J.H., Theobalt, C.: Efficient learning-based image enhancement: application to compression artifact removal and super-resolution. Technical Report MPI-I-2011-4-002, Max-Planck-Insitut für Informatik (February 2011)Google Scholar
  27. 27.
    Frank, M., Plaue, M., Rapp, K., Köthe, U., Jähne, B., Hamprecht, F.: Theoretical and experimental error analysis of continuous-wave time-of-flight range cameras. Optical Engineering 48(1), 13602 (2009)CrossRefGoogle Scholar
  28. 28.
    Lenzen, F., Schäfer, H., Garbe, C.: Denoising time-of-flight data with adaptive total variation. In: Bebis, G. (ed.) ISVC 2011, Part I. LNCS, vol. 6938, pp. 337–346. Springer, Heidelberg (2011)Google Scholar
  29. 29.
    Schöner, H., Moser, B., Dorrington, A.A., Payne, A., Cree, M.J., Heise, B., Bauer, F.: A clustering based denoising technique for range images of time of flight cameras. In: CIMCA/IAWTIC/ISE 2008, pp. 999–1004 (2008)Google Scholar
  30. 30.
    Moser, B., Bauer, F., Elbau, P., Heise, B., Schöner, H.: Denoising techniques for raw 3D data of ToF cameras based on clustering and wavelets. In: Proc. SPIE, vol. 6805 (2008)Google Scholar
  31. 31.
    Frank, M., Plaue, M., Hamprecht, F.A.: Denoising of continuous-wave time-of-flight depth images using confidence measures. Optical Engineering 48 (2009)Google Scholar
  32. 32.
    Edeler, T.: Bildverbesserung von Time-Of-Flight-Tiefenkarten. Shaker Verlag (2011)Google Scholar
  33. 33.
    Edeler, T., Ohliger, K., Hussmann, S., Mertins, A.: Time-of-flight depth image denoising using prior noise information. In: Proceedings ICSP, pp. 119–122 (2010)Google Scholar
  34. 34.
    Seitel, A., dos Santos, T.R., Mersmann, S., Penne, J., Groch, A., Yung, K., Tetzlaff, R., Meinzer, H.P., Maier-Hein, L.: Adaptive bilateral filter for image denoising and its application to in-vitro time-of-flight data, 796423–796423–8 (2011)Google Scholar
  35. 35.
    Schöner, H., Bauer, F., Dorrington, A., Heise, B., Wieser, V., Payne, A., Cree, M.J., Moser, B.: Image processing for 3d-scans generated by time of flight range cameras. SPIE Journal of Electronic Imaging 2 (2012)Google Scholar
  36. 36.
    Schuon, S., Theobalt, C., Davis, J., Thrun, S.: Lidarboost: Depth superresolution for tof 3d shape scanning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 343–350. IEEE (2009)Google Scholar
  37. 37.
    Mure-Dubois, J., Hügli, H., et al.: Fusion of time of flight camera point clouds. Workshop on Multi-camera and Multi-Modal Sensor Fusion Algorithms and Applications-M2SFA2 2008 (2008)Google Scholar
  38. 38.
    Edeler, T., Ohliger, K., Hussmann, S., Mertins, A.: Super resolution of time-of-flight depth images under consideration of spatially varying noise variance. In: 16th IEEE Int. Conf. on Image Processing (ICIP), Cairo, Egypt, pp. 1185–1188 (November 2009)Google Scholar
  39. 39.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S., et al.: A noise-aware filter for real-time depth upsampling. In: Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications-M2SFA2 2008 (2008)Google Scholar
  40. 40.
    Huhle, B., Schairer, T., Jenke, P., Straßer, W.: Robust non-local denoising of colored depth data. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2008, pp. 1–7. IEEE (2008)Google Scholar
  41. 41.
    Yeo, D., Kim, J., Baig, M.W., Shin, H., et al.: Adaptive bilateral filtering for noise removal in depth upsampling. In: 2010 International SoC Design Conference (ISOCC), pp. 36–39. IEEE (2010)Google Scholar
  42. 42.
    Park, J., Kim, H., Tai, Y.W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3d-tof cameras. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 1623–1630. IEEE (2011)Google Scholar
  43. 43.
    Reynolds, M., Doboš, J., Peel, L., Weyrich, T., Brostow, G.J.: Capturing time-of-flight data with confidence. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, pp. 945–952. IEEE (2011)Google Scholar
  44. 44.
    Mac Aodha, O., Campbell, N.D.F., Nair, A., Brostow, G.J.: Patch based synthesis for single depth image super-resolution. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 71–84. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  45. 45.
    Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: ACM SIGGRAPH 2007 Papers. SIGGRAPH 2007. ACM, New York (2007)Google Scholar
  46. 46.
    Eisemann, E., Durand, F.: Flash photography enhancement via intrinsic relighting. ACM Transactions on Graphics (TOG) 23, 673–678 (2004)CrossRefGoogle Scholar
  47. 47.
    Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM Transactions on Graphics (TOG) 23, 664–672 (2004)CrossRefGoogle Scholar
  48. 48.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision 40(1), 120–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Schmidt, M.: Analysis, Modeling and Dynamic Optimization of 3D Time-of-Flight Imaging Systems. Dissertation, IWR, Fakultät f ür Physik und Astronomie, Univ. Heidelberg (2011)Google Scholar
  50. 50.
    Blomgren, P., Chan, T.F.: Color tv: Total variation methods for restoration of vector-valued images. IEEE Transactions on Image Processing 7(3), 304–309 (1998)CrossRefGoogle Scholar
  51. 51.
    Schäfer, H., Lenzen, F., Garbe, C.S.: Depth and intensity based edge detection in time-of-flight images. In: Proceedings of 3DV. IEEE (in press, 2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Frank Lenzen
    • 1
    • 2
  • Kwang In Kim
    • 3
  • Henrik Schäfer
    • 1
    • 2
  • Rahul Nair
    • 1
    • 2
  • Stephan Meister
    • 1
    • 2
  • Florian Becker
    • 1
  • Christoph S. Garbe
    • 1
    • 2
  • Christian Theobalt
    • 3
    • 2
  1. 1.Heidelberg Collaboratory for Image Processing (HCI)Heidelberg UniversityHeidelbergGermany
  2. 2.Intel Visual Computing InstituteSaarland UniversitySaarbrückenGermany
  3. 3.Max-Planck-Institut für InformatikSaarland UniversitySaarbrückenGermany

Personalised recommendations