Temporally Consistent Disparity and Optical Flow via Efficient Spatio-temporal Filtering

  • Asmaa Hosni
  • Christoph Rhemann
  • Michael Bleyer
  • Margrit Gelautz
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7087)

Abstract

This paper presents a new efficient algorithm for computing temporally consistent disparity maps from video footage. Our method is motivated by recent work [1] that achieves high quality stereo results by smoothing disparity costs with a fast edge-preserving filter. This previous approach was designed to work with single static image pairs and does not maintain temporal coherency of disparity maps when applied to video streams.

The main contribution of our work is to transfer this concept to the spatio-temporal domain in order to efficiently achieve temporally consistent disparity maps, where disparity changes are aligned with spatio-temporal edges of the video sequence. We further show that our method can be used as spatio-temporal regularizer for optical flow estimation. Our approach can be implemented efficiently, achieving real-time results for stereo matching. Quantitative and qualitative results demonstrate that our approach (i) considerably improves over frame-by-frame methods for both stereo and optical flow; and (ii) outperforms the state-of-the-art for local space-time stereo approaches.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast Cost-Volume Filtering for Visual Correspondence and Beyond. In: CVPR (2011)Google Scholar
  2. 2.
    Yoon, K.J., Kweon, I.S.: Locally Adaptive Support-Weight Approach for Visual Correspondence Search. In: CVPR (2005)Google Scholar
  3. 3.
    Hosni, A., Bleyer, M., Gelautz, M., Rhemann, C.: Local Stereo Matching Using Geodesic Support Weights. In: ICIP (2009)Google Scholar
  4. 4.
    Hosni, A., Bleyer, M., Gelautz, M.: Near Real-Time Stereo With Adaptive Support Weight Approaches. In: 3DPVT (2010)Google Scholar
  5. 5.
    He, K., Sun, J., Tang, X.: Guided Image Filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 1–14. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  6. 6.
    Richardt, C., Orr, D., Davies, I., Criminisi, A., Dodgson, N.A.: Real-time Spatiotemporal Stereo Matching Using the Dual-Cross-Bilateral Grid. In: Daniilidis, K. (ed.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 510–523. Springer, Heidelberg (2010)Google Scholar
  7. 7.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1/2/3), 7–42 (2002), http://www.middlebury.edu/stereo/ CrossRefMATHGoogle Scholar
  8. 8.
    Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-Dimensional Scene Flow. In: ICCV (1999)Google Scholar
  9. 9.
    Leung, C., Appleton, B., Lovell, B.C., Sun, C.: An Energy Minimisation Approach to Stereo-Temporal Dense Reconstruction. In: ICPR (2004)Google Scholar
  10. 10.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: CVPR (2010)Google Scholar
  11. 11.
    Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: CVPR (2010)Google Scholar
  12. 12.
    Salgado, A., Sánchez, J.: Temporal Constraints in Large Optical Flow Estimation. In: Moreno Díaz, R., Pichler, F., Quesada Arencibia, A. (eds.) EUROCAST 2007. LNCS, vol. 4739, pp. 709–716. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  13. 13.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High Accuracy Optical Flow Estimation Based on a Theory for Warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  14. 14.
    Weickert, J., Schnörr, C.: Variational optic flow computation with a spatio-temporal smoothness constraint. In: JMIV, vol. 14, pp. 245–255 (2001)Google Scholar
  15. 15.
    Black, M.J., Anandan, P.: Robust dynamic motion estimation over time. In: CVPR (1991)Google Scholar
  16. 16.
    Nagel, H.H.: Extending the ’Oriented Smoothness Constraint’ into the Temporal Domain and the Estimation of Derivatives of Optical Flow. In: Faugeras, O. (ed.) ECCV 1990. LNCS, vol. 427, pp. 139–148. Springer, Heidelberg (1990)CrossRefGoogle Scholar
  17. 17.
    Zhang, L., Curless, B., Seitz, S.M.: Spacetime Stereo: Shape Recovery for Dynamic Scenes. In: CVPR (2003)Google Scholar
  18. 18.
    Zhang, L., Snavely, N., Curless, B., Seitz, S.M.: Spacetime faces: high-resolution capture for modeling and animation. In: SIGGRAPH (2004)Google Scholar
  19. 19.
    Davis, J., Nehab, D., Ramamoorthi, R., Rusinkiewicz, S.: Spacetime stereo: a unifying framework for depth from triangulation. In: PAMI, vol. 27(2), pp. 296–302 (2005)Google Scholar
  20. 20.
    Jenkin, M., Tsotsos, J.: Applying temporal constraints to the dynamic stereo problem. In: CVGIP, vol. 33, pp. 16–32 (1986)Google Scholar
  21. 21.
    Williams, O., Isard, M., MacCormick, J.: Estimating Disparity and Occlusions in Stereo Video Sequences. In: CVPR (2005)Google Scholar
  22. 22.
    Larsen, E.S., Mordohai, P., Pollefeys, M., Fuchs, H.: Temporally consistent reconstruction from multiple video streams using enhanced belief propagation. In: ICCV (2007)Google Scholar
  23. 23.
    Bleyer, M., Gelatuz, M.: Temporally Consistent Disparity Maps from Uncalibrated Stereo Videos. In: ISPA (2009)Google Scholar
  24. 24.
    Zimmer, H., Bruhn, A., Weickert, J.: Optic Flow in Harmony. In: IJCV, vol. 93, pp. 368 – 388 (2011)Google Scholar
  25. 25.
    Sizintsev, M., Wildes, R.P.: Spatiotemporal stereo via spatiotemporal quadric element (stequel) matching. In: CVPR (2009)Google Scholar
  26. 26.
    Steinbrücker, F., Pock, T., Cremers, D.: Large displacement optical flow computation without warping. In: ICCV (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Asmaa Hosni
    • 1
  • Christoph Rhemann
    • 1
  • Michael Bleyer
    • 1
  • Margrit Gelautz
    • 1
  1. 1.Institute for Software Technology and Interactive SystemsVienna University of TechnologyViennaAustria

Personalised recommendations