Advertisement

Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation

  • Eddy Ilg
  • Tonmoy SaikiaEmail author
  • Margret Keuper
  • Thomas Brox
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11216)

Abstract

Occlusions play an important role in disparity and optical flow estimation, since matching costs are not available in occluded areas and occlusions indicate depth or motion boundaries. Moreover, occlusions are relevant for motion segmentation and scene flow estimation. In this paper, we present an efficient learning-based approach to estimate occlusion areas jointly with disparities or optical flow. The estimated occlusions and motion boundaries clearly improve over the state-of-the-art. Moreover, we present networks with state-of-the-art performance on the popular KITTI benchmark and good generic performance. Making use of the estimated occlusions, we also show improved results on motion segmentation and scene flow estimation.

Notes

Acknowledgements

We acknowledge funding by the EU Horizon2020 project TrimBot2020 and by Gala Sports, and donation of a GPU server by Facebook. Margret Keuper acknowledges funding by DFG grant KE 2264/1-1.

Supplementary material

474200_1_En_38_MOESM1_ESM.pdf (11.9 mb)
Supplementary material 1 (pdf 12177 KB)

Supplementary material 2 (avi 21944 KB)

References

  1. 1.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. PAMI 33(5), 898–916 (2011)CrossRefGoogle Scholar
  2. 2.
    Bailer, C., Varanasi, K., Stricker, D.: CNN-based patch matching for optical flow with thresholded hinge embedding loss. In: CVPR (2017)Google Scholar
  3. 3.
    Behl, A., Jafari, O.H., Mustikovela, S.K., Alhaija, H.A., Rother, C., Geiger, A.: Bounding boxes, segmentations and object coordinates: how important is recognition for 3D scene flow estimation in autonomous driving scenarios? In: International Conference on Computer Vision (ICCV) (2017)Google Scholar
  4. 4.
    Black, M.J., Fleet, D.J.: Probabilistic detection and tracking of motion boundaries. Int. J. Comput. Vis. 38(3), 231–245 (2000)CrossRefGoogle Scholar
  5. 5.
    Brox, T., Malik, J.: Object segmentation by long term analysis of point trajectories. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6315, pp. 282–295. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15555-0_21CrossRefGoogle Scholar
  6. 6.
    Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. PAMI 33(3), 500–513 (2011)CrossRefGoogle Scholar
  7. 7.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33783-3_44CrossRefGoogle Scholar
  8. 8.
    Cheng, J., Tsai, Y.H., Wang, S., Yang, M.H.: SegFlow: joint learning for video object segmentation and optical flow (2017)Google Scholar
  9. 9.
    Deng, Y., Yang, Q., Lin, X., Tang, X.: Stereo correspondence with occlusion handling in a symmetric patch-based graph-cuts model (2007)CrossRefGoogle Scholar
  10. 10.
    Dollár, P., Zitnick, C.L.: Structured forests for fast edge detection (2013)Google Scholar
  11. 11.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: IEEE International Conference on Computer Vision (ICCV) (2015)Google Scholar
  12. 12.
    Geiger, D., Ladendorf, B., Yuille, A.: Occlusions and binocular stereo. IJCV 14(3), 211–226 (1995)CrossRefGoogle Scholar
  13. 13.
    Gidaris, S., Komodakis, N.: Detect, replace, refine: deep structured prediction for pixel wise labeling. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  14. 14.
    Hirschmüller, H.: Stereo processing by semiglobal matching and mutual information. PAMI 30(2), 328–341 (2008)CrossRefGoogle Scholar
  15. 15.
    Huguet, F., Devernay, F.: A variational method for scene flow estimation from stereo sequences. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–7, October 2007.  https://doi.org/10.1109/ICCV.2007.4409000
  16. 16.
    Humayun, A., Aodha, O.M., Brostow, G.J.: Learning to find occlusion regions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  17. 17.
    Hur, J., Roth, S.: MirrorFlow: exploiting symmetries in joint optical flow and occlusion estimation (2017)Google Scholar
  18. 18.
    Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  19. 19.
    Ishikawa, H., Geiger, D.: Global optimization using embedded graphs (2000)Google Scholar
  20. 20.
    Jia, Z., Gallagher, A., Chen, T.: Learning boundaries with color and depth (2013)Google Scholar
  21. 21.
    Kendall, A., et al.: End-to-end learning of geometry and context for deep stereo regression (2017)Google Scholar
  22. 22.
    Keuper, M., Andres, B., Brox, T.: Motion trajectory segmentation via minimum cost multicuts. In: ICCV (2015)Google Scholar
  23. 23.
    Keuper, M.: Higher-order minimum cost lifted multicuts for motion segmentation. In: The IEEE International Conference on Computer Vision (ICCV), October 2017Google Scholar
  24. 24.
    Kolmogorov, V., Monasse, P., Tan, P.: Kolmogorov and Zabih’s graph cuts stereo matching algorithm. Image Process. Line 4, 220–251 (2014).  https://doi.org/10.5201/ipol.2014.97. https://hal-enpc.archivesouvertes.fr/hal-01074878CrossRefGoogle Scholar
  25. 25.
    Lei, P., Li, F., Todorovic, S.: Joint spatio-temporal boundary detection and boundary flow prediction with a fully convolutional siamese network. In: CVPR (2018)Google Scholar
  26. 26.
    Leordeanu, M., Sukthankar, R., Sminchisescu, C.: Efficient closed-form solution to generalized boundary detection. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 516–529. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33765-9_37CrossRefGoogle Scholar
  27. 27.
    Leordeanu, M., Zanfir, A., Sminchisescu, C.: Locally affine sparse-to-dense matching for motion and occlusion estimation. In: IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
  28. 28.
    Luo, W., Schwing, A.G., Urtasun, R.: Efficient deep learning for stereo matching (2016)Google Scholar
  29. 29.
    Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  30. 30.
    Ochs, P., Malik, J., Brox, T.: Segmentation of moving objects by long term video analysis. IEEE TPAMI 36(6), 1187–1200 (2014)CrossRefGoogle Scholar
  31. 31.
    Pang, J., Sun, W., Ren, J.S.J., Yang, C., Yan, Q.: Cascade residual learning: a two-stage convolutional neural network for stereo matching. In: IEEE International Conference on Computer Vision (ICCV) Workshop (2017)Google Scholar
  32. 32.
    Perez-Rua, J.M., Crivelli, T., Bouthemy, P., Perez, P.: Determining occlusions from space and time image reconstructions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  33. 33.
    Quiroga, J., Brox, T., Devernay, F., Crowley, J.: Dense semi-rigid scene flow estimation from RGBD images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 567–582. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_37. https://hal.inria.fr/hal-01021925CrossRefGoogle Scholar
  34. 34.
    Ranjan, A., Black, M.: Optical flow estimation using a spatial pyramid network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  35. 35.
    Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: EpicFlow: edge-preserving interpolation of correspondences for optical flow. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  36. 36.
    Schuster, R., Bailer, C., Wasenmüller, O., Stricker, D.: Combining stereo disparity and optical flow for basic scene flow. In: Berns, K. (ed.) Commercial Vehicle Technology 2018, pp. 90–101. Springer, Wiesbaden (2018).  https://doi.org/10.1007/978-3-658-21300-8_8CrossRefGoogle Scholar
  37. 37.
    Schuster, R., Wasenmüller, O., Kuschk, G., Bailer, C., Stricker, D.: SceneFlowFields: dense interpolation of sparse scene flow correspondences. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2018)Google Scholar
  38. 38.
    Seki, A., Pollefeys, M.: SGM-Nets: semi-global matching with neural networks (2017)Google Scholar
  39. 39.
    Shaked, A., Wolf, L.: Improved stereo matching with constant highway networks and reflective confidence learning. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  40. 40.
    Sun, D., Liu, C., Pfister, H.: Local layering for joint motion estimation and occlusion detection (2014)Google Scholar
  41. 41.
    Sun, D., Liu, C., Pfister, H.: Local layering for joint motion estimation and occlusion detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  42. 42.
    Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: CVPR (2018)Google Scholar
  43. 43.
    Sundberg, P., Brox, T., Maire, M., Arbeláez, P., Malik, J.: Occlusion boundary detection and figure/ground assignment from optical flow. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  44. 44.
    Tan, P., Chambolle, A., Monasse, P.: Occlusion detection in dense stereo estimation with convex optimization. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 2543–2547, September 2017.  https://doi.org/10.1109/ICIP.2017.8296741
  45. 45.
    Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 475–480 (2005).  https://doi.org/10.1109/TPAMI.2005.63CrossRefGoogle Scholar
  46. 46.
    Vogel, C., Schindler, K., Roth, S.: Piecewise rigid scene flow. In: 2013 IEEE International Conference on Computer Vision, pp. 1377–1384, December 2013.  https://doi.org/10.1109/ICCV.2013.174
  47. 47.
    Wedel, A., Brox, T., Vaudrey, T., Rabe, C., Franke, U., Cremers, D.: Stereoscopic scene flow computation for 3D motion understanding. Int. J. Comput. Vis. 95(1), 29–51 (2011).  https://doi.org/10.1007/s11263-010-0404-0CrossRefzbMATHGoogle Scholar
  48. 48.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: ICCV - IEEE International Conference on Computer Vision, Sydney, Australia, pp. 1385–1392. IEEE, December 2013.  https://doi.org/10.1109/ICCV.2013.175, https://hal.inria.fr/hal-00873592
  49. 49.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: Learning to detect motion boundaries. In: CVPR 2015 - IEEE Conference on Computer Vision & Pattern Recognition. Boston, United States, June 2015. https://hal.inria.fr/hal-01142653
  50. 50.
    Xu, J., Ranftl, R., Koltun, V.: Accurate optical flow via direct cost volume processing. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  51. 51.
    Yamaguchi, K., McAllester, D., Urtasun, R.: Efficient joint segmentation, occlusion labeling, stereo and flow estimation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 756–771. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_49CrossRefGoogle Scholar
  52. 52.
    Zbontar, J., LeCun, Y.: Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 17(1–32), 2 (2016)zbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Eddy Ilg
    • 1
  • Tonmoy Saikia
    • 1
    Email author
  • Margret Keuper
    • 1
  • Thomas Brox
    • 1
  1. 1.University of FreiburgFreiburg im BreisgauGermany

Personalised recommendations