Depth Enhancement by Fusion for Passive and Active Sensing

  • Frederic Garcia
  • Djamila Aouada
  • Hashim Kemal Abdella
  • Thomas Solignac
  • Bruno Mirbach
  • Björn Ottersten
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7585)


This paper presents a general refinement procedure that enhances any given depth map obtained by passive or active sensing. Given a depth map, either estimated by triangulation methods or directly provided by the sensing system, and its corresponding 2-D image, we correct the depth values by separately treating regions with undesired effects such as empty holes, texture copying or edge blurring due to homogeneous regions, occlusions, and shadowing. In this work, we use recent depth enhancement filters intended for Time-of-Flight cameras, and adapt them to alternative depth sensing modalities, both active using an RGB-D camera and passive using a dense stereo camera. To that end, we propose specific masks to tackle areas in the scene that require a special treatment. Our experimental results show that such areas are satisfactorily handled by replacing erroneous depth measurements with accurate ones.


depth enhancement data fusion passive sensing active sensing 


  1. 1.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press (2003)Google Scholar
  2. 2.
    Salvi, J., Fernandez, S., Pribanic, T., Llado, X.: A state of the art in structured light patterns for surface profilometry. Pattern Recognition 43, 2666–2680 (2010)zbMATHCrossRefGoogle Scholar
  3. 3.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A noise-aware filter for real-time depth upsampling. In: Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications, ECCVW (2008)Google Scholar
  4. 4.
    Garcia, F., Mirbach, B., Ottersten, B., Grandidier, F., Cuesta, A.: Pixel Weighted Average Strategy for Depth Sensor Data Fusion. In: International Conference on Image Processing (ICIP), pp. 2805–2808 (2010)Google Scholar
  5. 5.
    Garcia, F., Aouada, D., Mirbach, B., Solignac, T., Ottersten, B.: A New Multi-lateral Filter for Real-Time Depth Enhancement. In: Advanced Video and Signal-Based Surveillance, AVSS (2011)Google Scholar
  6. 6.
    Min, D., Lu, J., Minh, N.D.: Depth Video Enhancement Based on Weighted Mode Filtering. IEEE Transactions on Image Processing (TIP) 21, 1176–1190 (2012)CrossRefGoogle Scholar
  7. 7.
    Kopf, J., Cohen, M., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: SIGGRAPH 2007: ACM SIGGRAPH 2007 papers, p. 96. ACM, New York (2007)CrossRefGoogle Scholar
  8. 8.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV, pp. 839–846 (1998)Google Scholar
  9. 9.
    Crabb, R., Tracey, C., Puranik, A., Davis, J.: Real-time foreground segmentation via range and color imaging. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–5 (2008)Google Scholar
  10. 10.
    Benhimane, S., Malis, E.: Real-time image-based tracking of planes using efficient second-order minimization. In: IEEE IROS, vol. 1, pp. 943–948 (2004)Google Scholar
  11. 11.
    Hirschmuller, H.: Real-time correlation-based stereo vision with reduced border errors. IJCV 47(1/2/3), 229–246 (2002)CrossRefGoogle Scholar
  12. 12.
    Yang, Q., Tan, K.H., Ahuja, N.: Real-time O(1) bilateral filtering. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 557–564 (2009)Google Scholar
  13. 13.
    Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. International Journal of Computer Vision 81, 24–52 (2009)CrossRefGoogle Scholar
  14. 14.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47, 7–42 (2002)zbMATHCrossRefGoogle Scholar
  15. 15.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. In: IEEE TIP, vol. 13-4, pp. 600–612 (2004)Google Scholar
  16. 16.
    Middlebury Stereo Datasets (2011),
  17. 17.
    Point Grey Research, Inc. (2011),

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Frederic Garcia
    • 1
  • Djamila Aouada
    • 1
  • Hashim Kemal Abdella
    • 1
    • 2
  • Thomas Solignac
    • 3
  • Bruno Mirbach
    • 3
  • Björn Ottersten
    • 1
  1. 1.Interdisciplinary Centre for Security, Reliability and TrustUniverstity of LuxembourgLuxembourg
  2. 2.Université de BourgogneFrance
  3. 3.Advanced Engineering - IEE S.A.Luxembourg

Personalised recommendations