Advertisement

Machine Vision and Applications

, Volume 28, Issue 3–4, pp 267–282 | Cite as

Image-guided ToF depth upsampling: a survey

  • Iván Eichhardt
  • Dmitry Chetverikov
  • Zsolt Jankó
Original Paper

Abstract

Recently, there has been remarkable growth of interest in the development and applications of time-of-flight (ToF) depth cameras. Despite the permanent improvement of their characteristics, the practical applicability of ToF cameras is still limited by low resolution and quality of depth measurements. This has motivated many researchers to combine ToF cameras with other sensors in order to enhance and upsample depth images. In this paper, we review the approaches that couple ToF depth images with high-resolution optical images. Other classes of upsampling methods are also briefly discussed. Finally, we provide an overview of performance evaluation tests presented in the related studies.

Keywords

ToF cameras Depth images Optical images Depth upsampling Survey 

Notes

Acknowledgements

We are grateful to Zinemath Zrt for providing test data. This research was supported in part by the programme ‘Highly industrialised region on the west part of Hungary with limited R&D capacity: Research and development programs related to strengthening the strategic future oriented industries manufacturing technologies and products of regional competences carried out in comprehensive collaboration’ of the Hungarian National Research, Development and Innovation Fund (NKFIA), Grant #VKSZ_12-1-2013-0038. This work was also supported by the NKFIA Grant #K-120233.

References

  1. 1.
    Alenya, G., Dellen, B., Torras, C.: 3D modelling of leaves from color and ToF data for robotized plant measuring. In: IEEE International Conference on Robotics and Automation, pp. 3408–3414 (2011)Google Scholar
  2. 2.
    Awate, S.P., Whitaker, R.T.: Higher-order image statistics for unsupervised, information-theoretic, adaptive, image filtering. Proceedings of Conference on Computer Vision and Pattern Recognition 2, 44–51 (2005)Google Scholar
  3. 3.
    Balure, C.S., Kini, M.R.: Depth image super-resolution: a review and wavelet perspective. In: International Conference on Computer Vision and Image Processing, pp. 543–555 (2017)Google Scholar
  4. 4.
    Barash, D.: Fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation. IEEE Trans. Pattern Anal. Mach. Intell. 24, 844–847 (2002)CrossRefGoogle Scholar
  5. 5.
    Bartczak, B., Koch, R.: Dense depth maps from low resolution time-of-flight depth and high resolution color views. In: Advances in Visual Computing, pp. 228–239. Springer, Heidelberg (2009)Google Scholar
  6. 6.
    Beder, C., Bartczak, B., Koch, R.: A comparison of PMD-cameras and stereo-vision for the task of surface reconstruction using patchlets. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  7. 7.
    Bevilacqua, A., Di Stefano, L., Azzari, P.: People tracking using a time-of-flight depth sensor. In: Proceedings of International Conference on Video and Signal Based Surveillance (2006)Google Scholar
  8. 8.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3, 492–526 (2010)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Buades, A., Coll, B., Morel, J.-M.: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4, 490–530 (2005)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Carter, J., Schmid, K., Waters, K., Betzhold, L., Hadley, B., Mataosky, R., Halleran, J.: Lidar 101: an introduction to lidar technology, data, and applications. Technical report, NOAA Coastal Services Center, Charleston, USA (2012)Google Scholar
  11. 11.
    Čech, J., Šara, R.: Efficient sampling of disparity space for fast and accurate matching. In: BenCOS Workshop, CVPR, pp. 1–8 (2007)Google Scholar
  12. 12.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A noise-aware filter for real-time depth upsampling. In: Proceedings of the ECCV Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications (2008)Google Scholar
  13. 13.
    Choi, J., Min, D., Ham, B., Sohn, K.: Spatial and temporal up-conversion technique for depth video. In: Proceedings of International Conference on Image Processing, pp. 3525–3528 (2009)Google Scholar
  14. 14.
    Choi, O., Lim, H., Kang, B., Kim, Y.S., Lee, K., Kim, J.D.K., Kim, C.-Y.: Discrete and continuous optimizations for depth image super-resolution. In: Proceedings of IS&T/SPIE Electronic, Imaging, pp. 82900C–82900C (2012)Google Scholar
  15. 15.
    Cui, Y., Schuon, S., Chan, D., Thrun, S., Theobalt, C.: 3D shape scanning with a time-of-flight camera. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1173–1180 (2010)Google Scholar
  16. 16.
    De Cubber, G., Doroftei, D., Sahli, H., Baudoin, Y.: Outdoor terrain traversability analysis for robot navigation using a time-of-flight camera. In: Proceedings of RGB-D Workshop on 3D Perception in Robotics (2011)Google Scholar
  17. 17.
    Dellen, B., Alenyà, R., Sergi Foix, S., Torras, C.: 3D object reconstruction from Swissranger sensor data using a spring-mass model. In: Proceedings of International Conference on Computer Vision Theory and Applications, vol. 2, pp. 368–372 (2009)Google Scholar
  18. 18.
    Diebel, J., Thrun, S.: An application of Markov random fields to range sensing. In: Proceedings of Advances in Neural Information Processing Systems, pp. 291–298 (2005)Google Scholar
  19. 19.
    Dolson, J., Baek, J., Plagemann, C., Thrun, S.: Upsampling range data in dynamic environments. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1141–1148 (2010)Google Scholar
  20. 20.
    Eichhardt, I., Jankó, Z., Chetverikov, D.: Novel methods for image-guided ToF depth upsampling. In: IEEE International Conference on Systems, Man, and Cybernetics, pp. 002073–002078 (2016)Google Scholar
  21. 21.
    Einramhof, P., Olufs, M., Vincze, S.: Experimental evaluation of state of the art 3D-sensors for mobile robot navigation. In: Proceedings of Austrian Association for Pattern Recognition Workshop, pp. 153–160 (2007)Google Scholar
  22. 22.
    Falie, D., Buzuloiu, V.: Wide range time of flight camera for outdoor surveillance. In: Proceedings of IEEE Symposium on Microwaves, Radar and Remote Sensing, , pp. 79–82 (2008)Google Scholar
  23. 23.
    Fattal, R.: Image upsampling via imposed edge statistics. ACM Tran. Graphics 26, 95 (2007)CrossRefGoogle Scholar
  24. 24.
    Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: Proceedings of International Conference on Computer Vision, pp. 993–1000 (2013)Google Scholar
  25. 25.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Fofi, D., Sliwa, T., Voisin, Y.: A comparative survey on invisible structured light. Electron. Imaging 2004, 90–98 (2004)Google Scholar
  27. 27.
    Foix, S., Alenya, G., Andrade-Cetto, J., Torras, C.: Object modeling using a ToF camera under an uncertainty reduction approach. In: Proceedings of International Conference on Robotics and Automation, pp. 1306–1312 (2010)Google Scholar
  28. 28.
    Foix, S., Alenya, G., Torras, C.: Lock-in time-of-flight (ToF) cameras: a survey. Sensors J. 11(9), 1917–1926 (2011)CrossRefGoogle Scholar
  29. 29.
    Foix, S., Alenyà, R., Torras, C.: Exploitation of time-of-flight (ToF) cameras. Technical Report IRI-DT-10-07, IRI-UPC (2010)Google Scholar
  30. 30.
    Fu, M., Zhou, W.: Depth map super-resolution via extended weighted mode filtering. Vis. Commun. Image Process. 1, 1–4 (2016)Google Scholar
  31. 31.
    Fuchs, S., May, S.: Calibration and registration for precise surface reconstruction with time-of-flight cameras. Int. J. Intell. Syst. Technol. Appl. 5, 274–284 (2008)Google Scholar
  32. 32.
    Fukushima, N., Takeuchi, K., Kojima, A.: Self-similarity matching with predictive linear upsampling for depth map. In: 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video, pp. 1–4 (2016)Google Scholar
  33. 33.
    Garcia, F., Mirbach, B., Ottersten, B., Grandidier, F., Cuesta, A.: Pixel weighted average strategy for depth sensor data fusion. In: Proceedings of International Conference on Image Processing, pp. 2805–2808 (2010)Google Scholar
  34. 34.
    Gemeiner, P., Jojic, P., Vincze, M.: Selecting good corners for structure and motion recovery using a time-of-flight camera. In: International Conference on Intelligent Robots and Systems, pp. 5711–5716 (2009)Google Scholar
  35. 35.
    Gokturk, S.B., Tomasi, C.: 3D head tracking based on recognition and interpolation using a time-of-flight depth sensor. Proceedings of Conference on Computer Vision and Pattern Recognition 2, 211–217 (2004)Google Scholar
  36. 36.
    Gong, M., Yang, L., Wang, R., Gong, M.: A performance study on different cost aggregation approaches used in real-time stereo matching. Int. J. Comput. Vis. 75, 283–296 (2007)CrossRefGoogle Scholar
  37. 37.
    Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.): Time-of-Flight and Depth Imaging, Sensors, Algorithms, and Applications. Springer, Heidelberg (2013)Google Scholar
  38. 38.
    Guan, H., Li, J., Yu, Y., Chapman, M., Wang, C.: Automated road information extraction from mobile laser scanning data. IEEE Trans. Intell. Transp. Syst. 16, 194–205 (2015)CrossRefGoogle Scholar
  39. 39.
    Guomundsson, S.A., Aanæs, H., Larsen, R.: Fusion of stereo vision and time-of-flight imaging for improved 3D estimation. Int. J. Intell. Syst. Technol. Appl. 5(3), 425–433 (2008)Google Scholar
  40. 40.
    Guomundsson, S.A., Larsen, R., Aanæs, H., Pardas, M., Casas, J.R.: ToF imaging in smart room environments towards improved people tracking. In: Proceedings of Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–6 (2008)Google Scholar
  41. 41.
    Hahne, U., Alexa, M.: Combining time-of-flight depth and stereo images without accurate extrinsic calibration. Int. J. Intell. Syst. Technol. Appl. 5, 325–333 (2008)Google Scholar
  42. 42.
    Hahne, U., Alexa, M.: Depth imaging by combining time-of-flight and on-demand stereo. In: Kolb, A., Koch, R. (eds.) Dynamic 3D Imaging, pp. 70–83. Springer, Berlin (2009)CrossRefGoogle Scholar
  43. 43.
    Hahne, U., Alexa, M.: Exposure fusion for time-of-flight imaging. In: Computer Graphics Forum, vol. 30, pp. 1887–1894. Wiley, London (2011)Google Scholar
  44. 44.
    Han, Y., Lee, J.-Y., Kweon, I.: High quality shape from a single RGD-D image under uncalibrated natural illumination. In: Proceedings of International Conference on Computer Vision, pp. 1617–1624 (2013)Google Scholar
  45. 45.
    Hansard, M., Lee, S., Choi, O., Horaud, R.: Time-of-Flight Cameras. Springer, Berlin (2013)CrossRefGoogle Scholar
  46. 46.
    Harrison, A., Newman, P.: Image and sparse laser fusion for dense scene reconstruction. In: Howard, A., Iagnemma, K., Kelly, A. (eds.) Field and Service Robotics, pp. 219–228. Springer, Berlin (2010)CrossRefGoogle Scholar
  47. 47.
    He, K., Sun, J., Tang, X.: Guided image filtering. In: Proceedings of European Conference on Computer Vision, pp. 1–14 (2010)Google Scholar
  48. 48.
    Heidelberg Collaboratory for Image Processing, Ruprecht-Karl University: Time of flight stereo fusion collection. http://hci.iwr.uni-heidelberg.de/Benchmarks/ (2016)
  49. 49.
    Herbort, S., Wöhler, C.: An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods. 3D Res. 2(3), 1–17 (2011)Google Scholar
  50. 50.
    Herrera, C., Kannala, J., Heikkilä, J.: Joint depth and color camera calibration with distortion correction. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2058–2064 (2012)CrossRefGoogle Scholar
  51. 51.
    Ho, Y.-S., Kang, Y.-S.: Multi-view depth generation using multi-depth camera system. In: International Conference on 3D Systems and Application, pp. 67–70 (2010)Google Scholar
  52. 52.
    Hornacek, M., Rhemann, C., Gelautz, M., Rother, C.: Depth super resolution by rigid body self-similarity in 3D. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1123–1130 (2013)Google Scholar
  53. 53.
    Huhle, B., Schairer, T., Jenke, P., Straßer, W.: Fusion of range and color images for denoising and resolution enhancement with a non-local filter. Comput. Vis. Image Underst. 114, 1336–1345 (2010)CrossRefGoogle Scholar
  54. 54.
    Hui, T.-W., Loy, C.C., Tang, X.: Depth map super-resolution by deep multi-scale guidance. In: European Conference on Computer Vision, pp. 353–369 (2016)Google Scholar
  55. 55.
    Hussmann, S., Liepert, T.: Robot vision system based on a 3D-ToF camera. In: Proceedings of Conference on Instrumentation and Measurement Technology, pp. 1–5 (2007)Google Scholar
  56. 56.
    Ihrke, I., Kutulakos, K.N., Lensch, H.P.A., Magnor, M., Heidrich, W.: State of the art in transparent and specular object reconstruction. In: EUROGRAPHICS 2008 State of the Art Reports (2008)Google Scholar
  57. 57.
    Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Hodges, S., Kohli, P., Shotton, J., Davison, A.J., Fitzgibbon, A.: KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of ACM Symposium on User Interface Software and Technology, pp. 559–568 (2011)Google Scholar
  58. 58.
    Jung, J., Lee, J.-Y., Jeong, Y., Kweon, I.: Time-of-flight sensor calibration for a color and depth camera pair. PAMI 37, 1501–1513 (2015)CrossRefGoogle Scholar
  59. 59.
    Kamilov, U.S., Boufounos, P.T.: Depth superresolution using motion adaptive regularization. In: IEEE International Conference on Multimedia & Expo Workshops, pp. 1–6 (2016)Google Scholar
  60. 60.
    Kang, Y.-S., Ho, Y.-S.: High-quality multi-view depth generation using multiple color and depth cameras. In: IEEE International Conference on Multimedia and Expo, pp. 1405–1410 (2010)Google Scholar
  61. 61.
    Katz, S., Adler, A., Yahav, G.: Combined depth filtering and super resolution. US Patent 8,660,362. www.google.com/patents/US8660362 (2014)
  62. 62.
    Kim, C., Yu, H., Yang, G.: Depth super resolution using bilateral filter. In: Proceedings of International Congress on Image and Signal Processing, vol. 2, pp. 1067–1071(2011)Google Scholar
  63. 63.
    Kim, Y.M., Theobalt, C., Diebel, J., Kosecka, J., Miscusik, B., Thrun, S.: Multi-view image and ToF sensor fusion for dense 3D reconstruction. In: ICCV Workshops, pp. 1542–1549 (2009)Google Scholar
  64. 64.
    Knoop, S., Vacek, S., Dillmann, R.: Sensor fusion for 3D human body tracking with an articulated 3D body model. In: Proceedings of International Conference on Robotics and Automation, pp. 1686–1691 (2006)Google Scholar
  65. 65.
    Kolb, A., Barth, E., Koch, R., Larsen, R.: Time-of-flight cameras in computer graphics. Comput. Graphics Forum 29, 141–159 (2010)CrossRefGoogle Scholar
  66. 66.
    Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. ACM Trans. Graphics 26, 673–678 (2007)Google Scholar
  67. 67.
    Kuhnert, K.-D., Stommel, M.: Fusion of stereo-camera and PMD-camera data for real-time suited precise 3D environment reconstruction. In: Proceedings of International Conference on Intelligent Robots and Systems, pp. 4780–4785 (2006)Google Scholar
  68. 68.
    Kuznetsova, A.: A ToF camera calibration toolbox. http://github.com/kapibara/ToF-Calibration (2015)
  69. 69.
    Kuznetsova, A., Rosenhahn, B.: On calibration of a low-cost time-of-flight camera. In: ECCV Workshop on Consumer Depth Cameras for Computer Vision (2014)Google Scholar
  70. 70.
    Lai, K., Bo, L., Ren, X., Fox, D.: A large-scale hierarchical multi-view RGB-D object dataset. In: IEEE International Conference on Robotics and Automation, pp. 1817–1824 (2011)Google Scholar
  71. 71.
    Langmann, B., Hartmann, K., Loffeld, O.: Comparison of depth super-resolution methods for 2D/3D images. Int. J. Comput. Inf. Syst. Ind. Manag. Appl. 3, 635–645 (2011)Google Scholar
  72. 72.
    Lefloch, D., Nair, R., Lenzen, F., Schäfer, H., Streeter, L., Cree, M.J., Koch, R., Kolb, A.: Technical foundation and calibration methods for time-of-flight cameras. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, pp. 3–24. Springer, Berlin (2013)Google Scholar
  73. 73.
    Li, J., Lu, Z., Zeng, G., Gan, R., Wang, L., Zha, H.: A joint learning-based method for multi-view depth map super resolution. In: Proceedings of Asian Conference on Pattern Recognition, pp. 456–460 (2013)Google Scholar
  74. 74.
    Li, J., Zeng, G., Gan, R., Zha, H., Wang, L.: A Bayesian approach to uncertainty-based depth map super resolution. In: Proceedings of Asian Conference on Computer Vision, pp. 205–216 (2012)Google Scholar
  75. 75.
    Li, L.: Time-of-flight camera—an introduction. Technical report SLOA190B, Texas Instruments, 2014. www.ti.com/lit/wp/sloa190b/sloa190b.pdf
  76. 76.
    Li, Y., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep joint image filtering. In: European Conference on Computer Vision, pp. 154–169 (2016)Google Scholar
  77. 77.
    Lindner, M., Schiller, I., Kolb, A., Koch, R.: Time-of-flight sensor calibration for accurate range sensing. Comput. Vis. Image Underst. 114, 1318–1328 (2010)CrossRefGoogle Scholar
  78. 78.
    Liu, M., Zhao, Y., Liang, J., Lin, C., Bai, H., Yao, C.: Depth map up-sampling with fractal dimension and texture-depth boundary consistencies. Neurocomputing (2017). doi: 10.1016/j.neucom.2016.11.067
  79. 79.
    Liu, X., Fujimura, K.: Hand gesture recognition using depth data. In: Proceedings of International Conference on Automatic Face and Gesture Recognition, pp. 529–534 (2004)Google Scholar
  80. 80.
    Lo, K.-H. , Wang, Y.-C., Hua, K.-L.: Edge-preserving depth map upsampling by joint trilateral filter. IEEE Trans. Cybern. pp. 1–14 (2017). doi: 10.1109/TCYB.2016.2637661
  81. 81.
    Lu, J., Min, D., Pahwa, R.S., Do, M.N.: A revisit to MRF-based depth map super-resolution and enhancement. In: International Conference on Acoustics, Speech and Signal Processing, pp. 985–988 (2011)Google Scholar
  82. 82.
    Lu, S., Ren, X., Liu, F.: Depth enhancement via low-rank matrix completion. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 3390–3397 (2014)Google Scholar
  83. 83.
    Mac Aodha, O., Campbell, N.D.F., Nair, A., Brostow, G.J.: Patch based synthesis for single depth image super-resolution. In: Proceedings of European Conference on Computer Vision, pp. 71–84 (2012)Google Scholar
  84. 84.
    Mandal, S., Bhavsar, A., Sao, A.K.: Depth map restoration from undersampled data. IEEE Trans. Image Process. 26, 119–134 (2017)MathSciNetCrossRefGoogle Scholar
  85. 85.
    May, S., Droeschel, D., Holz, D., Wiesen, C., Fuchs, S.: 3D pose estimation and mapping with time-of-flight cameras. In: Proceedings of IROS Workshop on 3D Mapping (2008)Google Scholar
  86. 86.
    Milanfar, P.: A tour of modern image filtering: new insights and methods, both practical and theoretical. IEEE Signal Process. Mag. 30, 106–128 (2013)CrossRefGoogle Scholar
  87. 87.
    Min, D., Lu, J., Do, M.N.: Depth video enhancement based on weighted mode filtering. IEEETIP 21, 1176–1190 (2012)MathSciNetGoogle Scholar
  88. 88.
    Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT press, Cambridge (2012)MATHGoogle Scholar
  89. 89.
    Nair, R., Meister, S., Lambers, M., Balda, M., Hofmann, H., Kolb, A., Kondermann, D., Jähne, B.: Ground truth for evaluating time of flight imaging. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, pp. 52–74. Springer, Berlin (2013)Google Scholar
  90. 90.
    Nair, R., Ruhl, K., Lenzen, F., Meister, S., Schäfer, H., Garbe, C.S., Eisemann, M., Magnor, M., Kondermann, D.: A survey on time-of-flight stereo fusion. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, pp. 105–127. Springer, Berlin (2013)Google Scholar
  91. 91.
    Nanda, H., Fujimura, K.: Visual tracking using depth data. In: Proceedings of Conference on Computer Vision and Pattern Recognition Workshops (2004)Google Scholar
  92. 92.
    Nehab, D., Rusinkiewicz, S., Davis, J., Ramamoorthi, R.: Efficiently combining positions and normals for precise 3D geometry. ACM Trans. Graphics 24, 536–543 (2005)CrossRefGoogle Scholar
  93. 93.
    Newcombe, R.A., Davison, A.J., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136 (2011)Google Scholar
  94. 94.
    Nguyen, C.V., Izadi, S., Lovell, D.: Modeling kinect sensor noise for improved 3D reconstruction and tracking. In: Second Joint 3DIM/3DPVT Conference: 3D Imaging, Modeling, Processing, Visualization & Transmission, pp. 524–530 (2012)Google Scholar
  95. 95.
    Or-El, R., Rosman, G., Wetzler, A., Kimmel, R., Bruckstein, A.M.: RGBD-fusion: real-time high precision depth recovery. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 5407–5416 (2015)Google Scholar
  96. 96.
    Paris, S., Kornprobst, P., Tumblin, J., Durand, F.: Bilateral Filtering. Now Publishers Inc., Norwell (2009)MATHGoogle Scholar
  97. 97.
    Park, J., Kim, H., Tai, Y.-W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3D-ToF cameras. In: Proceedings of International Conference on Computer Vision, pp. 1623–1630 (2011)Google Scholar
  98. 98.
    Park, J., Kim, H., Tai, Y.-W., Brown, M.S., Kweon, I.: High-quality depth map upsampling and completion for RGB-D cameras. IEEE Trans. Image Process. 23, 5559–5572 (2014)MathSciNetCrossRefGoogle Scholar
  99. 99.
    Pfeifer, N., Lichti, D., Böhm, J., Karel, W.: 3D cameras: errors, calibration and orientation. In: Remondino, F., Stoppa, D. (eds.) TOF Range-Imaging Cameras, pp. 117–138. Springer, Berlin (2013)CrossRefGoogle Scholar
  100. 100.
    Porikli, F.: Constant time O(1) bilateral filtering. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)Google Scholar
  101. 101.
    Prusak, A., Melnychuk, O., Roth, H., Schiller, I.: Pose estimation and map building with a time-of-flight-camera for robot navigation. Int. J. Intell. Syst. Technol. Appl. 5, 355–364 (2008)Google Scholar
  102. 102.
    Remondino, F., Stoppa, D.: ToF Range-Imaging Cameras. Springer, Berlin (2013)CrossRefGoogle Scholar
  103. 103.
    Richardt, C., Stoll, C., Dodgson, N.A., Seidel, H.-P., Theobalt, C.: Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. Comput. Graphics Forum 31, 247–256 (2012)CrossRefGoogle Scholar
  104. 104.
    Riegler, G., Rüther, M., Bischof, H.: ATGV-net: accurate depth super-resolution. In: European Conference on Computer Vision, pp. 268–284 (2016)Google Scholar
  105. 105.
    Riemens, A.K., Gangwal, O.P., Barenbrug, B., Berretty, R.-P.M.: Multistep joint bilateral depth upsampling. In: IS&T/SPIE Electronic Imaging, pp. 72570M–72570M (2009)Google Scholar
  106. 106.
    Robot Vision Laboratory, Graz University of Technology: ToFMark—Depth upsampling evaluation dataset. http://rvlab.icg.tugraz.at/tofmark/ (2014)
  107. 107.
    Rotman, D., Gilboa, G.: A depth restoration occlusionless temporal dataset. In: International Conference on 3D Vision, pp. 176–184 (2016)Google Scholar
  108. 108.
    Ruiz-Sarmiento, J.R., Galindo, C., Gonzalez, J.: Improving human face detection through ToF cameras for ambient intelligence applications. In: Ambient Intelligence-Software and Applications, pp. 125–132. Springer, Berlin (2011)Google Scholar
  109. 109.
    Salvi, J., Fernandez, S., Pribanic, T., Llado, X.: A state of the art in structured light patterns for surface profilometry. Pattern Recogn. 43, 2666–2680 (2010)CrossRefMATHGoogle Scholar
  110. 110.
    Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nesic, N., Wang, X., Westling, P.: Middlebury stereo datasets. http://vision.middlebury.edu/stereo/data/ (2001–2014)
  111. 111.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002)CrossRefMATHGoogle Scholar
  112. 112.
    Schaul, L., Fredembach, C., Süsstrunk, S.: Color image dehazing using the near-infrared. In: Proceedings of International Conference on Image Processing, pp. 1629–1632 (2009)Google Scholar
  113. 113.
    Schmidt, M.: Analysis, Modeling and Dynamic Optimization of 3D Time-of-Flight Imaging Systems. PhD thesis, Ruperto-Carola University of Heidelberg, Germany (2011)Google Scholar
  114. 114.
    Schoenberg, J.R., Nathan, A., Campbell, M.: Segmentation of dense range information in complex urban scenes. In: Proceedings of International Conference on Intelligent Robots and Systems, pp. 2033–2038. IEEE (2010)Google Scholar
  115. 115.
    Schuon, S., Theobalt, C., Davis, J., Thrun, S.: Lidarboost: depth superresolution for ToF 3D shape scanning. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 343–350 (2009)Google Scholar
  116. 116.
    Schwarz, S., Olsson, R., Sjostrom, M.: Depth sensing for 3DTV: a survey. IEEE MultiMedia 20, 10–17 (2013)CrossRefGoogle Scholar
  117. 117.
    Schwarz, S., Sjöström, M., Olsson, R.: Multivariate sensitivity analysis of time-of-flight sensor fusion. 3D Res. 5, 1–16 (2014)Google Scholar
  118. 118.
    Schwarz, S., Sjöström, M., Olsson, R.: Temporal consistent depth map upscaling for 3DTV. In: IS&T/SPIE Electronic Imaging, pp. 901302–901302 (2014)Google Scholar
  119. 119.
    Schwarz, S., Sjöström, M., Olsson, R.: Weighted optimization approach to time-of-flight sensor fusion. IEEE Trans. Image Process. 23, 214–225 (2014)MathSciNetCrossRefGoogle Scholar
  120. 120.
    Schwarz, S.: Gaining Depth: Time-of-Flight Sensor Fusion for Three-Dimensional Video Content Creation. PhD thesis, Mittuniversitetet, Sweden (2014)Google Scholar
  121. 121.
    Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. Conference on Computer Vision and Pattern Recognition 1, 519–528 (2006)Google Scholar
  122. 122.
    Snavely, N., Zitnick, C.L., Kang, S.B., Cohen, M.: Stylizing 2.5-D video. In: Proceedings of 4th International Symposium on Non-photorealistic Animation and Rendering, pp. 63–69. ACM (2006)Google Scholar
  123. 123.
    Soh, Y., Sim, J.Y., Kim, C.S., Lee, S.U.: Superpixel-based depth image super-resolution. In: IS&T/SPIE Electronic Imaging, pp. 82900D–82900D. Int. Society for Optics and Photonics (2012)Google Scholar
  124. 124.
    Sonka, M., Hlavac, V., Boyle, R.: Image Processing, Analysis, and Machine Vision. Thomson, Stamford (2008)Google Scholar
  125. 125.
    Stoykova, E., Alatan, A.A., Benzie, P., Grammalidis, N., Malassiotis, S., Ostermann, J., Piekh, S.: 3-D time-varying scene capture technologies–a survey. IEEE Trans. Circuits Syst. 17, 1568–1586 (2007)Google Scholar
  126. 126.
    Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, Berlin (2010)MATHGoogle Scholar
  127. 127.
    Tallón, M., Babacan, S.D., Mateos, J., Do, M.N., Molina, R., Katsaggelos, A.K.: Upsampling and denoising of depth maps via joint-segmentation. In: Proceedings of European Signal Processing Conference, pp. 245–249 (2012)Google Scholar
  128. 128.
    Thielemann, J.T., Breivik, G.M., Berge, A.: Pipeline landmark detection for autonomous robot navigation using time-of-flight imagery. In: Proceedings of Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–7 (2008)Google Scholar
  129. 129.
    Tian, J., Ma, K.-K.: A survey on super-resolution imaging. Signal Image Video Process. 5, 329–342 (2011)CrossRefGoogle Scholar
  130. 130.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of International Conference on Computer Vision, pp. 839–846 (1998)Google Scholar
  131. 131.
    Van Ouwerkerk, J.D.: Image super-resolution survey. Image Vis. Comput. 24, 1039–1052 (2006)CrossRefGoogle Scholar
  132. 132.
    Villena-Martínez, V., Fuster-Guilló, A., Azorín-López, J., Saval-Calvo, M., Mora-Pascual, J., Garcia-Rodriguez, J., Garcia-Garcia, A.: A quantitative comparison of calibration methods for RGB-D sensors using different technologies. Sensors 17, 243 (2017)CrossRefGoogle Scholar
  133. 133.
    Vosters, L., Varekamp, C., de Haan, G.: Evaluation of efficient high quality depth upsampling methods for 3DTV. In: IS&T/SPIE Electronic Imaging, pp. 865005–865005 (2013)Google Scholar
  134. 134.
    Vosters, L., Varekamp, C., de Haan G.: Overview of efficient high-quality state-of-the-art depth enhancement methods by thorough design space exploration. J. Real-Time Image Process. 1–21 (2015). doi: 10.1007/s11554-015-0537-z
  135. 135.
    Weingarten, J.W., Gruener, G., Siegwart, R.: A state-of-the-art 3D sensor for robot navigation. Proceedings of International Conference on Intelligent Robots and Systems 3, 2155–2160 (2004)Google Scholar
  136. 136.
    Xiang, X., Li, G., Tong, J., Zhang, M., Pan, Z.: Real-time spatial and depth upsampling for range data. Trans. Comput. Sci. XII: Special Issue Cyberworlds 6670, 78 (2011)Google Scholar
  137. 137.
    Xu, X., Po, L.-M., Ng, K.-H., Feng, L., Cheung, K.-W., Cheung, C.-H., Ting, C.-W.: Depth map misalignment correction and dilation for DIBR view synthesis. Signal Process. Image Commun. 28, 1023–1045 (2013)CrossRefGoogle Scholar
  138. 138.
    Yang, K., Dou, Y., Chen, X., Lv, S., Qiao, P.: Depth enhancement via non-local means filter. In: International Conference on Advanced Computational Intelligence, pp. 126–130 (2015)Google Scholar
  139. 139.
    Yang, Q., Ahuja, N., Yang, R., Tan, K.-H., Davis, J., Culbertson, B., Apostolopoulos, J., Wang, G.: Fusion of median and bilateral filtering for range image upsampling. IEEE Trans. Image Process. 22, 4841–4852 (2013)MathSciNetCrossRefGoogle Scholar
  140. 140.
    Yang, Q., Tan, K.-H., Ahuja. N.: Real-time O(1) bilateral filtering. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 557–564 (2009)Google Scholar
  141. 141.
    Yang, Q., Tan, K.H., Culbertson, B., Apostolopoulos, J.: Fusion of active and passive sensors for fast 3D capture. In: Proceedings of IEEE International Workshop on Multimedia Signal Processing, pp. 69–74 (2010)Google Scholar
  142. 142.
    Yang, Q., Yang, R., Davis, J., Nistér, D.: Spatial-depth super resolution for range images. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  143. 143.
    Yin, L., Yang, R., Gabbouj, M., Neuvo, Y.: Weighted median filters: a tutorial. IEEE Trans. Circuits Syst. II Analog. Digital Signal Process 43, 157–192 (1996)Google Scholar
  144. 144.
    Yu, L.-F., Yeung, S.-K., Tai, Y-W., Lin. S.: Shading-based shape refinement of RGBD images. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1415–1422 (2013)Google Scholar
  145. 145.
    Yuan, F., Swadzba, A., Philippsen, R., Engin, O., Hanheide, M., Wachsmuth, S.: Laser-based navigation enhanced with 3D time-of-flight data. In: Proceedings of International Conference on Robotics and Automation, pp. 2844–2850 (2009)Google Scholar
  146. 146.
    Yuan, L., Jin, X., Yuan. C.: Enhanced joint trilateral up-sampling for super-resolution. In: Pacific Rim Conference on Multimedia, pp. 518–526 (2016)Google Scholar
  147. 147.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1330–1334 (2000)CrossRefGoogle Scholar
  148. 148.
    Zhang, Z.: Microsoft Kinect sensor and its effect. IEEE MultiMedia 19, 4–10 (2012)CrossRefGoogle Scholar
  149. 149.
    Zhu, J., Wang, L., Gao, J., Yang, R.: Spatial-temporal fusion for high accuracy depth maps using dynamic MRFs. IEEE Trans. Pattern Anal. Mach. Intell. 32, 899–909 (2010)CrossRefGoogle Scholar
  150. 150.
    Zhu, J., Wang, L., Yang, R., Davis. J.E.: Fusion of time-of-flight depth and stereo for high accuracy depth maps. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)Google Scholar
  151. 151.
    Zhu, J., Wang, L., Yang, R., Davis, J.E.: Pan. Z.: Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1400–1414 (2011)CrossRefGoogle Scholar
  152. 152.
    Zinemath, Z. The zLense platform. www.zinemath.com/ (2014)

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Iván Eichhardt
    • 1
    • 2
  • Dmitry Chetverikov
    • 1
    • 2
  • Zsolt Jankó
    • 2
  1. 1.Eötvös Loránd UniversityBudapestHungary
  2. 2.MTA SZTAKIBudapestHungary

Personalised recommendations