Advertisement

Real-time stereo using approximated joint bilateral filtering and dynamic programming

  • 899 Accesses

  • 17 Citations

Abstract

We present a stereo algorithm that is capable of estimating scene depth information with high accuracy and in real time. The key idea is to employ an adaptive cost-volume filtering stage in a dynamic programming optimization framework. The per-pixel matching costs are aggregated via a separable implementation of the bilateral filtering technique. Our separable approximation offers comparable edge-preserving filtering capability and leads to a significant reduction in computational complexity compared to the traditional 2D filter. This cost aggregation step resolves the disparity inconsistency between scanlines, which are the typical problem for conventional dynamic programming based stereo approaches. Our algorithm is driven by two design goals: real-time performance and high accuracy depth estimation. For computational efficiency, we utilize the vector processing capability and parallelism in commodity graphics hardware to speed up this aggregation process over two orders of magnitude. Over 90 million disparity evaluations per second [the number of disparity evaluations per seconds (MDE/s) corresponds to the product of the number of pixels and the disparity range and the obtained frame rate and, therefore, captures the performance of a stereo algorithm in a single number] are achieved in our current implementation. In terms of quality, quantitative evaluation using data sets with ground truth disparities shows that our approach is one of the state-of-the-art real-time stereo algorithms.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Notes

  1. 1.

    It is important to note that some previous methods are not implemented on the same processors. The MDE/s numbers reported in this table do not reflect a fair comparison across different platforms, but only a reference for the readers.

References

  1. 1.

    Ansar, A., Castano, A., Matthies L.: Enhanced real-time stereo using bilateral filtering. In: Proceedings of the International Symposium on 3D Data Processing, Visualization and Transmission (2004)

  2. 2.

    Bobick, A.F., Intille, S.S.: Large occlusion stereo. Int. J. Comput. Vis. 33(2), 181–200 (1999)

  3. 3.

    Boykov, Y., Veksler, O., Zabih, R.: A variable window approach to early vision. IEEE Trans. Pattern Anal. Mach. Intell. 20(12), 1283–1294 (1998)

  4. 4.

    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)

  5. 5.

    Chen, J., Paris, S., Durand F.: Real-time edge-aware image processing with the bilateral grid. In: Proceedings of the ACM Siggraph Conference (2007)

  6. 6.

    Cox, I.J., Hingorani, S.L., Rao, S.B., Maggs, B.M.: A maximum likelihood stereo algorithm. Comput. Vis. Image Underst. 63(3), 542–567 (1996)

  7. 7.

    Deng, Y., Lin, X.: A fast line segment based dense stereo algorithm using tree dynamic programming. In: Proceedings of the European Conference on Computer Vision (2006)

  8. 8.

    Durand, F., Dorsey, J.: Fast bilateral filtering for the display of highdynamic-range images. In: Proceedings of the ACM Siggraph Conference (2002)

  9. 9.

    Egnal, G., Wildes, R.P.: Detecting binocular half-occlusions: empirical comparisons of five approaches. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1127–1133 (2002)

  10. 10.

    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70(1), 41–54 (2006)

  11. 11.

    Forstmann, S., Ohya, J., Kanou, Y., Schmitt, A., Thuering, S.: Real-time stereo by using dynamic programming. In: Proceedings of CVPR Workshop on Real-time 3D Sensors and Their Use (2004)

  12. 12.

    Geiger, D., Ladendorf, B., Yuille, A.: Occlusions and binocular stereo. Int. J. Comput. Vis. 14(3), 211–226 (1995)

  13. 13.

    Georgoulas, C., Andreadis, I.: A real-time fuzzy hardware structure for disparity map computation. J. Real-Time Image Process. 6(4), 257–273 (2011)

  14. 14.

    Gong, M., Yang, R., Wang, L., Gong, M.: A performance study on different cost aggregation approaches used in real-time stereo matching. Int. J. Comput. Vis. 75(2), 283–296 (2007)

  15. 15.

    Gong, M., Yang, Y.-H.: Near real-time reliable stereo matching using programmable graphics hardware. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2005)

  16. 16.

    Gupta, R.K., Cho, S.-Y.: Real-time stereo matching using adaptive binary window. In: Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (2010)

  17. 17.

    Gupta, R.K., Cho, S.-Y.: A correlation-based approach for real-time stereo matching. In: Proceedings of International Conference on Advances in Visual Computing (2010)

  18. 18.

    Gutemberg, G.-F.: An optimal time space algorithm for dense stereo matching. J. Real-Time Image Process. 7(2), 69–86 (2012)

  19. 19.

    He K., Sun J., and Tang X.: Guided image filtering. In: Proceedings of European Conference on Computer Vision (2010)

  20. 20.

    Kanade, T., Okutomi, M.: A stereo matching algorithm with an adaptive window: theory and experiment. IEEE Trans. Pattern Anal. Mach. Intell. 16(9), 920–932 (1994)

  21. 21.

    Kanade, T., Yoshida, A., Oda, K., Kano, H., Tanaka M.: Stereo method for video-rate dense depth mapping and its new applications. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (1996)

  22. 22.

    Kang, S.B., Szeliski, R., Chai, J.: Handling occlusions in dense multi-view stereo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2001)

  23. 23.

    Kim, J.C., Lee, K.M., Choi, B.T., Lee, S.U.: A dense stereo matching using two-pass dynamic programming with generalized ground control points. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2005)

  24. 24.

    Kimura, S., Shinbo, T., Yamaguchi, H., Kawamura, E., Naka, K.: A convolver-based real-time stereo machine (sazan). In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (1999)

  25. 25.

    Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: Proceedings of ACM Siggraph Conference (2007)

  26. 26.

    Lei, C., Selzer, J., Yang, Y.-H.: Region-tree based stereo using dynamic programming optimization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2006)

  27. 27.

    Mattoccia, S., Giardino, S., Gambini, A.: Accurate and efficient cost aggregation strategy for stereo correspondence based on approximated joint bilateral filtering. In: Proceedings of Asian Conference on Computer Vision (2009)

  28. 28.

    Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J.Comput. Vis. 81(1), 24–52 (2009)

  29. 29.

    Paris, S., Kornprobst, P., Tumblin, J., Durand, F.: Bilateral filtering: theory and applications. Found. Trends Comput. Graph. Vis. 16(9), 1–73 (2009)

  30. 30.

    Pham, T.Q., van Vliet L.J.: Separable bilateral filtering for fast video preprocessing. In: Proceedings of IEEE International Conference on Multimedia and Expo (2005)

  31. 31.

    Digiclops trinocular stereovision system. http://www.ptgrey.com/products/triclopsSDK/triclops.pdf

  32. 32.

    Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2010)

  33. 33.

    Richardt, C., Orr, D., Davies, I., Criminisi, A., Dodgson, N.: Real-time spatiotemporal stereo matching using the dual-cross-bilateral grid. In: Proceedings of European Conference on Computer Vision (2010)

  34. 34.

    Salmen, J., Schlipsing, M., Edelbrunner, J., Hegemann, S., Lueke, S.: Real-time stereo vision: making more out of dynamic programming. In: Proceedings of International Conference on Computer Analysis of Images and Patterns (2009)

  35. 35.

    Scharstein, D., Szeliski, R. http://vision.middlebury.edu/stereo/

  36. 36.

    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1), 7–42 (2002)

  37. 37.

    Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2003)

  38. 38.

    Smith, B.M., Zhang, L., Jin, H.: Stereo matching with nonparametric priors in feature space. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2009)

  39. 39.

    Sun, C.: Fast stereo matching using rectangular subregioning and 3D maximum-surface techniques. Int. J. Comput. Vis. 47(1–3), 99–177 (2002)

  40. 40.

    Sun, J., Zheng, N.-N., Shum, H.-Y.: Stereo matching using belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 25(7), 787–800 (2003)

  41. 41.

    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Proceedings of IEEE International Conference on Computer Vision (1998)

  42. 42.

    Tombari, F., Mattoccia, S., Stefano, L.D., Addimanda, E.: Classification and performance evaluation of different aggregation costs for stereo matching. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2008)

  43. 43.

    Veksler, O.: Fast variable window for stereo correspondence using integral images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2003)

  44. 44.

    Veksler, O.: Stereo correspondence by dynamic programming on a tree. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2005)

  45. 45.

    Wang, L., Liao, M., Gong, M., Yang, R., Nister, D.: High-quality real-time stereo using adaptive cost aggregation and dynamic programming. In: Proceedings of International Symposium on 3D Data Processing, Visualization and Transmission (2006)

  46. 46.

    Weiss, B.: Fast median and bilateral filtering. In: Proceedings of ACM Siggraph Conference (2006)

  47. 47.

    Yang, Q., Engels, C., Akbarzadeh, A.: Near real-time stereo for weakly-textured scenes. In: Proceedings of British Machine Vision Conference (2008)

  48. 48.

    Yang, Q., Wang, L., Yang, R., Wang, S., Liao, M. Nister, D.: Real-time global stereo matching using hierarchical belief propagation. In: Proceedings of British Machine Vision Conference (2006)

  49. 49.

    Yang, R., Pollefeys, M.: Multi-resolution real-time stereo on commodity graphics hardware. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2003).

  50. 50.

    Yang, R., Pollefeys, M., Yang, H., Welch, G.: A unified approach to real-time, multi-resolution, multi-baseline 2D view synthesis and 3D depth estimation using commodity graphics hardware. Int. J. Image Graph. 4(4), 1–25 (2003)

  51. 51.

    Yoon, K.-J., Kweon, I.-S.: Adaptive support-weight approach for correspondence search. IEEE Trans. Pattern Anal. Mach. Intell. 28(4), 650–656 (2006)

  52. 52.

    Yu, W., Chen, T., Franchetti, F., Hoe, J.C.: High performance stereo vision designed for massively data parallel platforms. IEEE Trans. Circuits Syst. Video Technol.20(11), 1509–1519 (2009)

  53. 53.

    Zhang, K., Lu, J., Lafruit, G., Lauwereins, R., Van Gool, L.: Real-time accurate stereo with bitwise fast voting on cuda. In: Proceedings of IEEE Workshop on Embedded Computer Vision (2009)

Download references

Acknowledgments

The authors would like to thank the editor and all the reviewers for their constructive comments. This work is supported in part by University of Kentucky Research Foundation, US National Science Foundation award IIS-0448185, CPA-0811647, MRI-0923131, National Science Foundation of China grant No. 60872069, and Zhejiang Provincial Natural Science Foundation of China grant 2011C11053.

Author information

Correspondence to Liang Wang.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material (AVI 8.62 MB)

Supplementary material (AVI 8.62 MB)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Wang, L., Yang, R., Gong, M. et al. Real-time stereo using approximated joint bilateral filtering and dynamic programming. J Real-Time Image Proc 9, 447–461 (2014). https://doi.org/10.1007/s11554-012-0275-4

Download citation

Keywords

  • Real-time stereo
  • Cost aggregation
  • Bilateral filtering
  • Dynamic programming
  • Disparity map
  • Stereo video