Skip to main content

Illumination-Aware Large Displacement Optical Flow

  • Conference paper
  • First Online:
Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2017)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10746))

  • 1055 Accesses

Abstract

The integration of feature matches for handling large displacements is one of the key concepts of recent variational optical flow methods. In this context, many existing approaches rely on confidence measures to identify locations where a poor initial match can potentially be improved by adaptively integrating flow proposals. One very intuitive confidence measure to identify such locations is the matching cost of the data term. Problems arise, however, in the presence of illumination changes, since brightness constancy does not hold and invariant constancy assumptions typically discard too much information for an identification of poor matches. In this paper, we suggest a pipeline approach that addresses the aforementioned problem in two ways. First, we propose a novel confidence measure based on the illumination-compensated brightness constancy assumption. By estimating illumination changes from a pre-computed flow this measure allows us to reliably identify poor matches even in the presence of varying illumination. Secondly, in contrast to many existing pipeline approaches, we propose to integrate only feature matches that have been obtained from dense variational methods. This in turn not only provides robust matches due to the inherent regularization, it also demonstrates that in many cases sparse descriptor matches are not needed for large displacement optical flow. Experiments on the Sintel benchmark and on common large displacement sequences demonstrate the benefits of our strategy. They show a clear improvement over the baseline method and a comparable performance as similar methods from the literature based on sparse feature matches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Alvarez, L., EsclarĂ­n, J., LefĂ©bure, M., SĂ¡nchez, J.: A PDE model for computing the optical flow. In: Proceedings of Congreso de Ecuaciones Diferenciales y Aplicaciones, pp. 1349–1356 (1999)

    Google Scholar 

  2. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2010)

    Article  Google Scholar 

  3. Barnes, C., Shechtman, E., Goldman, D.B., Finkelstein, A.: The generalized PatchMatch correspondence algorithm. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 29–43. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15558-1_3

    Chapter  Google Scholar 

  4. Berg, A., Malik, J.: Geometric blur for template matching. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 607–614 (2001)

    Google Scholar 

  5. Black, M.J., Anandan, P.: Robust dynamic motion estimation over time. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 292–302 (1991)

    Google Scholar 

  6. Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)

    Article  Google Scholar 

  7. Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  8. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Chapter  Google Scholar 

  9. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)

    Article  Google Scholar 

  10. Bruhn, A., Weickert, J.: A confidence measure for variational optic flow methods. In: Klette, R., Kozera, R., Noakes, L., Weickert, J. (eds.) Geometric Properties from Incomplete Data, Computational Imaging and Vision, vol. 31, pp. 283–297. Springer, Dordrecht (2006). https://doi.org/10.1007/1-4020-3858-8_15

    Chapter  Google Scholar 

  11. Charbonnier, P., Blanc-Féraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of IEEE International Conference on Image Processing, pp. 168–172 (1994)

    Google Scholar 

  12. Chen, Z., Jin, H., Lin, Z., Cohen, S., Wu, Y.: Large displacement optical flow from nearest neighbor fields. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2443–2450 (2013)

    Google Scholar 

  13. Dalal, N., Triggs, B.: Histogram of oriented gradients for human detection. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 886–893 (2005)

    Google Scholar 

  14. Demetz, O., Stoll, M., Volz, S., Weickert, J., Bruhn, A.: Learning brightness transfer functions for the joint recovery of illumination changes and optical flow. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 455–471. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_30

    Google Scholar 

  15. Drayer, B., Brox, T.: Combinatorial regularization of descriptor matching for optical flow estimation. In: Proceedings of British Machine Vision Conference, pp. 42.1–42.12 (2015)

    Google Scholar 

  16. Förstner, W., GĂ¼lch, E.: A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proceedings of ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data, pp. 281–305 (1987)

    Google Scholar 

  17. Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    Article  Google Scholar 

  18. Lempitsky, V., Roth, S., Rother, C.: FusionFlow: discrete-continuous optimization for optical flow estimation. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

    Google Scholar 

  19. Lowe, D., Bruckstein, A.M., Kimmel, R.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  20. Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W.T.: SIFT flow: dense correspondence across different scenes. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 28–42. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88690-7_3

    Chapter  Google Scholar 

  21. Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8, 565–593 (1986)

    Article  Google Scholar 

  22. Nir, T., Bruckstein, A.M., Kimmel, R.: Over-parameterized variational optical flow. Int. J. Comput. Vis. 76(2), 205–216 (2008)

    Article  Google Scholar 

  23. Rashwan, H.A., Mohamed, M.A., García, M.A., Mertsching, B., Puig, D.: Illumination robust optical flow model based on histogram of oriented gradients. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 354–363. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40602-7_38

    Chapter  Google Scholar 

  24. Sevilla-Lara, L., Sun, D., Learned-Miller, E.G., Black, M.J.: Optical flow estimation with channel constancy. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 423–438. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_28

    Google Scholar 

  25. Stoll, M., Volz, S., Bruhn, A.: Variational large displacement optical flow without feature matches. In: Pelillo, M., Hancock, E. (eds.) EMMCVPR 2017. LNCS, vol. 10746, pp. 79–92. Springer, Cham (2017)

    Google Scholar 

  26. Stoll, M., Volz, S., Bruhn, A.: Adaptive integration of feature matches into variational optical flow methods. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7726, pp. 1–14. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37431-9_1

    Chapter  Google Scholar 

  27. Stoll, M., Volz, S., Maurer, D., Bruhn, A.: A time-efficient optimisation framework for parameters of optical flow methods. In: Sharma, P., Bianchi, F.M. (eds.) SCIA 2017. LNCS, vol. 10269, pp. 41–53. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59126-1_4

    Chapter  Google Scholar 

  28. Tu, Z., Poppe, R., Veltkamp, R.C.: Weighted local intensity fusion method for variational optical flow estimation. Pattern Recogn. 50, 223–232 (2016)

    Article  Google Scholar 

  29. Volz, S., Bruhn, A., Valgaerts, L., Zimmer, H.: Modeling temporal coherence for optical flow. In: Proceedings of International Conference on Computer Vision, pp. 1116–1123 (2011)

    Google Scholar 

  30. Weickert, J., Schnörr, C.: A theoretical framework for convex regularizers in PDE-based computation of image motion. Int. J. Comput. Vis. 45(3), 245–264 (2001)

    Article  MATH  Google Scholar 

  31. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: Proceedings of International Conference on Computer Vision, pp. 1385–1392 (2013)

    Google Scholar 

  32. Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1744–1757 (2012)

    Article  Google Scholar 

  33. Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., Seidel, H.-P.: Complementary optic flow. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 207–220. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03641-5_16

    Chapter  Google Scholar 

Download references

Acknowledgements

We thank the German Research Foundation (DFG) for financial support within project B04 of SFB/Transregio 161.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Michael Stoll .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Stoll, M., Maurer, D., Volz, S., Bruhn, A. (2018). Illumination-Aware Large Displacement Optical Flow. In: Pelillo, M., Hancock, E. (eds) Energy Minimization Methods in Computer Vision and Pattern Recognition. EMMCVPR 2017. Lecture Notes in Computer Science(), vol 10746. Springer, Cham. https://doi.org/10.1007/978-3-319-78199-0_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-78199-0_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-78198-3

  • Online ISBN: 978-3-319-78199-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics