Skip to main content
SpringerLink
Log in

Continual Occlusion and Optical Flow Estimation

  • Conference paper
  • First Online:
Computer Vision – ACCV 2018 (ACCV 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11364))

Included in the following conference series:

Abstract

Two optical flow estimation problems are addressed: (i) occlusion estimation and handling, and (ii) estimation from image sequences longer than two frames. The proposed ContinualFlow method estimates occlusions before flow, avoiding the use of flow corrupted by occlusions for their estimation. We show that providing occlusion masks as an additional input to flow estimation improves the standard performance metric by more than 25% on both KITTI and Sintel. As a second contribution, a novel method for incorporating information from past frames into flow estimation is introduced. The previous frame flow serves as an input to occlusion estimation and as a prior in occluded regions, i.e. those without visual correspondences. By continually using the previous frame flow, ContinualFlow performance improves further by 18% on KITTI and 7% on Sintel, achieving top performance on KITTI and Sintel.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    As of the submission date, July 7, 2018.

  2. 2.

    The “Final pass” category.

  3. 3.

    Excluding scene flow methods.

  4. 4.

    As of July 7, 2018.

References

  1. Bailer, C., Taetz, B., Stricker, D.: Flow fields: dense correspondence fields for highly accurate large displacement optical flow estimation. In: ICCV (2015)

    Google Scholar 

  2. Bailer, C., Varanasi, K., Stricker, D.: CNN-based patch matching for optical flow with thresholded hinge embedding loss. In: CVPR, pp. 3250–3259 (2017)

    Google Scholar 

  3. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. IJCV (2011). https://doi.org/10.1007/s11263-010-0390-2

  4. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: International Conference on Machine Learning, pp. 41–48. ACM (2009)

    Google Scholar 

  5. Black, M.J., Anandan, P.: Robust dynamic motion estimation over time. In: CVPR (1991)

    Google Scholar 

  6. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_44

    Chapter  Google Scholar 

  7. Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taixé, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: CVPR 2017. IEEE (2017)

    Google Scholar 

  8. Chen, Q., Koltun, V.: Full flow: optical flow estimation by global optimization over regular grids. In: CVPR (2016)

    Google Scholar 

  9. Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: ICCV, pp. 2758–2766, December 2015

    Google Scholar 

  10. Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: CVPR (2016)

    Google Scholar 

  11. Garg, R., Pizarro, L., Rueckert, D., Agapito, L.: Dense multi-frame optic flow for non-rigid objects using subspace constraints. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6495, pp. 460–473. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19282-1_37

    Chapter  Google Scholar 

  12. Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. IJCV 104(3), 286–314 (2013)

    Article  MathSciNet  Google Scholar 

  13. Güney, F., Geiger, A.: Deep discrete flow. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10114, pp. 207–224. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54190-7_13

    Chapter  Google Scholar 

  14. Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)

    Article  Google Scholar 

  15. Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely connected convolutional networks. In: CVPR (2017)

    Google Scholar 

  16. Hui, T.W., Tang, X., Loy, C.C.: LiteFlowNet: a lightweight convolutional neural network for optical flow estimation. In: CVPR, June 2018

    Google Scholar 

  17. Hur, J., Roth, S.: MirrorFlow: exploiting symmetries in joint optical flow and occlusion estimation. ICCV (2017)

    Google Scholar 

  18. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: CVPR (2017)

    Google Scholar 

  19. Irani, M.: Multi-frame correspondence estimation using subspace constraints. IJCV 48(3), 173–194 (2002)

    Article  MathSciNet  Google Scholar 

  20. Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: ANIPS (2015)

    Google Scholar 

  21. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  22. Kondermann, D., et al.: The HCI benchmark suite: stereo and flow ground truth with uncertainties for urban autonomous driving. In: CVPR (2016)

    Google Scholar 

  23. Maurer, D., Bruhn, A.: ProFlow: learning to predict optical flow. In: BMVC (2018)

    Google Scholar 

  24. Mayer, N., et al.: What makes good synthetic training data for learning disparity and optical flow estimation? IJCV 126(9), 942–960 (2018)

    Article  Google Scholar 

  25. Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: CVPR (2016)

    Google Scholar 

  26. Meister, S., Hur, J., Roth, S.: UnFlow: unsupervised learning of optical flow with a bidirectional census loss. In: AAAI (2018)

    Google Scholar 

  27. Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: CVPR (2015)

    Google Scholar 

  28. Murray, D.W., Buxton, B.F.: Scene segmentation from visual motion using global optimization. PAMI 9, 220–228 (1987)

    Article  Google Scholar 

  29. Pang, J., Sun, W., Ren, J.S., Yang, C., Yan, Q.: Cascade residual learning: a two-stage convolutional neural network for stereo matching. In: CVPR (2017)

    Google Scholar 

  30. Pérez, J.S., Meinhardt-Llopis, E., Facciolo, G.: TV-L1 optical flow estimation. Image Process. Line 3, 137–150 (2013)

    Article  Google Scholar 

  31. Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: CVPR (2018)

    Google Scholar 

  32. Robust vision challenge team: robust vision challenge (2018). http://www.robustvision.net. Accessed 8 July 2018

  33. Schuster, R., Bailer, C., Wasenmüller, O., Stricker, D.: FlowFields++: accurate optical flow correspondences meet robust interpolation. In: ICIP (2018)

    Google Scholar 

  34. Sun, D., Liu, C., Pfister, H.: Local layering for joint motion estimation and occlusion detection. In: CVPR, pp. 1098–1105 (2014)

    Google Scholar 

  35. Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: CVPR (2018)

    Google Scholar 

  36. Sundaram, N., Brox, T., Keutzer, K.: Dense point trajectories by GPU-accelerated large displacement optical flow. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 438–451. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15549-9_32

    Chapter  Google Scholar 

  37. Unger, M., Werlberger, M., Pock, T., Bischof, H.: Joint motion estimation and segmentation of complex scenes with label costs and occlusion modeling. In: CVPR, pp. 1878–1885. IEEE (2012)

    Google Scholar 

  38. Volz, S., Bruhn, A., Valgaerts, L., Zimmer, H.: Modeling temporal coherence for optical flow. In: ICCV (2011)

    Google Scholar 

  39. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  40. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: ICCV (2013)

    Google Scholar 

  41. Wulff, J., Sevilla-Lara, L., Black, M.J.: Optical flow in mostly rigid scenes. In: CVPR (2017)

    Google Scholar 

  42. Xiao, J., Cheng, H., Sawhney, H., Rao, C., Isnardi, M.: Bilateral filtering-based optical flow estimation with occlusion detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 211–224. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_17

    Chapter  Google Scholar 

  43. Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV, pp. 1395–1403 (2015)

    Google Scholar 

Download references

Acknowledgements

The research was supported by Toyota Motor Europe, CTU student grant SGS17/185/OHK3/3T/13 and the OP VVV MEYS project CZ.02.1.01/0.0/0.0/16_019/0000765 Research Center for Informatics.

Author information

Authors and Affiliations

  1. Center for Machine Perception, Faculty of Electrical Engineering, Czech Technical University in Prague, Prague, Czech Republic

    Michal Neoral, Jan Šochman & Jiří Matas

Authors
  1. Michal Neoral
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Šochman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiří Matas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michal Neoral .

Editor information

Editors and Affiliations

  1. IIIT Hyderabad, Hyderabad, India

    C.V. Jawahar

  2. ANU, Canberra, ACT, Australia

    Hongdong Li

  3. Simon Fraser University, Burnaby, BC, Canada

    Greg Mori

  4. ETH Zurich, Zurich, Zürich, Switzerland

    Konrad Schindler

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Neoral, M., Šochman, J., Matas, J. (2019). Continual Occlusion and Optical Flow Estimation. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11364. Springer, Cham. https://doi.org/10.1007/978-3-030-20870-7_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20870-7_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20869-1

  • Online ISBN: 978-3-030-20870-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation