Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation

  • Koichiro Yamaguchi
  • David McAllester
  • Raquel Urtasun
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8693)


In this paper we propose a slanted plane model for jointly recovering an image segmentation, a dense depth estimate as well as boundary labels (such as occlusion boundaries) from a static scene given two frames of a stereo pair captured from a moving vehicle. Towards this goal we propose a new optimization algorithm for our SLIC-like objective which preserves connecteness of image segments and exploits shape regularization in the form of boundary length. We demonstrate the performance of our approach in the challenging stereo and flow KITTI benchmarks and show superior results to the state-of-the-art. Importantly, these results can be achieved an order of magnitude faster than competing approaches.


Flow Estimation Stereo Pair Boundary Pixel Autonomous Driving Trifocal Tensor 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  2. 2.
    Basha, T., Moses, Y., Kiryati, N.: Multi-view scene flow estimation: A view centered variational approach. International journal of computer vision 101(1), 6–21 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Birchfield, S., Tomasi, C.: Multiway cut for stereo and motion with slanted surfaces. In: CVPR, vol. 1, pp. 489–495. IEEE (1999)Google Scholar
  4. 4.
    Bleyer, M., Gelautz, M.: A layered stereo matching algorithm using image segmentation and global visibility constraints. ISPRS Journal of Photogrammetry and Remote Sensing 59(3), 128–150 (2005)CrossRefGoogle Scholar
  5. 5.
    Braux-Zin, J., Dupont, R., Bartoli, A.: A general dense image matching framework combining direct and feature-based costs. In: ICCV (2013)Google Scholar
  6. 6.
    Costeira, J.P., Kanade, T.: A multibody factorization method for independently moving objects. International Journal of Computer Vision 29(3), 159–179 (1998)CrossRefGoogle Scholar
  7. 7.
    Dellaert, F., Seitz, S.M., Thorpe, C.E., Thrun, S.: Structure from motion without correspondence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 557–564. IEEE (2000)Google Scholar
  8. 8.
    Demetz, O., Hafner, D., Weickert, J.: The complete rank transform: A tool for accurate and morphologically invariant matching of structure. In: BMVC (2013)Google Scholar
  9. 9.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition, CVPR (2012)Google Scholar
  10. 10.
    Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003)Google Scholar
  11. 11.
    Hartley, R.I.: Lines and points in three views and the trifocal tensor. International Journal of Computer Vision 22(2), 125–140 (1997)CrossRefGoogle Scholar
  12. 12.
    Hermann, S., Klette, R.: Iterative semi-global matching for robust driver assistance systems. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part III. LNCS, vol. 7726, pp. 465–478. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  13. 13.
    Hirschmuller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 807–814. IEEE (2005)Google Scholar
  14. 14.
    Huguet, F., Devernay, F.: A variational method for scene flow estimation from stereo sequences. In: IEEE 11th International Conference on Computer Vision, ICCV 2007, pp. 1–7. IEEE (2007)Google Scholar
  15. 15.
    Klaus, A., Sormann, M., Karner, K.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In: 18th International Conference on Pattern Recognition, ICPR 2006, vol. 3, pp. 15–18. IEEE (2006)Google Scholar
  16. 16.
    Kuschk, G., Cremers, D.: Fast and accurate large-scale stereo reconstruction using variational methods. In: ICCV Workshop on Big Data in 3D Computer Vision (2013)Google Scholar
  17. 17.
    Rabe, C., Müller, T., Wedel, A., Franke, U.: Dense, robust, and accurate motion field estimation from stereo image sequences in real-time. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 582–595. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  18. 18.
    Ranftl, R., Pock, T., Bischof, H.: Minimizing TGV-based Variational Models with Non-Convex Data terms. In: ICSSVM (2013)Google Scholar
  19. 19.
    Rashwan, H.A., Mohamed, M.A., García, M.A., Mertsching, B., Puig, D.: Illumination robust optical flow model based on histogram of oriented gradients. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 354–363. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  20. 20.
    Scharstein, D., Szeliski, R.: Middlebury stereo vision page (2002),
  21. 21.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3d. ACM Transactions on Graphics (TOG) 25(3), 835–846 (2006)CrossRefGoogle Scholar
  22. 22.
    Spangenberg, R., Langner, T., Rojas, R.: Weighted semi-global matching and center-symmetric census transform for robust driver assistance. In: Wilson, R., Hancock, E., Bors, A., Smith, W. (eds.) CAIP 2013, Part II. LNCS, vol. 8048, pp. 34–41. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  23. 23.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. International Journal of Computer Vision 9(2), 137–154 (1992)CrossRefGoogle Scholar
  24. 24.
    Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision 1999, vol. 2, pp. 722–729. IEEE (1999)Google Scholar
  25. 25.
    Vogel, C., Roth, S., Schindler, K.: An evaluation of data costs for optical flow. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 343–353. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  26. 26.
    Vogel, C., Roth, S., Schindler, K.: Piecewise rigid scene flow. In: ICCV (2013)Google Scholar
  27. 27.
    Wang, Z.-F., Zheng, Z.-G.: A region based stereo matching algorithm using cooperative optimization. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
  28. 28.
    Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., Cremers, D.: Efficient dense scene flow from sparse or dense stereo data. Springer (2008)Google Scholar
  29. 29.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: Large displacement optical flow with deep matching. In: ICCV (2013)Google Scholar
  30. 30.
    Yamaguchi, K., Hazan, T., McAllester, D., Urtasun, R.: Continuous markov random fields for robust stereo estimation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 45–58. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  31. 31.
    Yamaguchi, K., McAllester, D., Urtasun, R.: Robust monocular epipolar flow estimation. In: CVPR (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Koichiro Yamaguchi
    • 1
  • David McAllester
    • 2
  • Raquel Urtasun
    • 3
  1. 1.Toyota Central R&D Labs., Inc.AichiJapan
  2. 2.Toyota Technological Institute at ChicagoUSA
  3. 3.University of TorontoCanada

Personalised recommendations