Skip to main content
SpringerLink
Log in

A Variational Aggregation Framework for Patch-Based Optical Flow Estimation

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

We propose a variational aggregation method for optical flow estimation. It consists of a two-step framework, first estimating a collection of parametric motion models to generate motion candidates, and then reconstructing a global dense motion field. The aggregation step is designed as a motion reconstruction problem from spatially varying sets of motion candidates given by parametric motion models. Our method is designed to capture large displacements in a variational framework without requiring any coarse-to-fine strategy. We handle occlusion with a motion inpainting approach in the candidates computation step. By performing parametric motion estimation, we combine the robustness to noise of local parametric methods with the accuracy yielded by global regularization. We demonstrate the performance of our aggregation approach by comparing it to standard variational methods and a discrete aggregation approach on the Middlebury and MPI Sintel datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Without loss of generality, isotropic circular patches could be considered as well.

  2. http://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/index.php.

  3. http://www.irisa.fr/vista/Motion2D/.

  4. http://www.mathworks.com/matlabcentral/fileexchange/23142-iterative-pyramidal-lk-optical-flow.

  5. http://www.irisa.fr/vista/Motion2D/.

  6. http://lmb.informatik.uni-freiburg.de/resources/software.php.

  7. http://gpu4vision.icg.tugraz.at/index.php?content=downloads.php.

  8. http://lmb.informatik.uni-freiburg.de/resources/software.php.

References

  1. Alba, A., Arce-Santana, E., Riviera, M.: Optical flow estimation with prior models obtained from phase correlation. In: Advances in Visual Computing, pp. 417–426. Springer, Berlin (2010)

  2. Arias, P., Facciolo, G., Caselles, V., Sapiro, G.: A variational framework for exemplar-based image inpainting. Int. J. Comput. Vis. 93(3), 319–347 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  3. Ayvaci, A., Raptis, M., Soatto, S.: Sparse occlusion detection with optical flow. Int. J. Comput. Vis. 97(3), 322–338 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bailer, C., Taetz, B., Stricker, D.: Flow fields: dense correspondence fields for highly accurate large displacement optical flow estimation. In: IEEE International Conference on Computer Vision, pp. 4015–4023 (2015)

  5. Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2011)

    Article  Google Scholar 

  6. Bao, L., Yang, Q., Jin, H.: Fast edge-preserving patchmatch for large displacement optical flow. In: Computer Vision and Pattern Recognition (CVPR), Columbus (2014)

  7. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24 (2009)

    Article  Google Scholar 

  8. Barron, J., Fleet, D., Beauchemin, S.: Evaluation of optical flow. Int. J. Comput. Vis. 12(1), 43–77 (1994)

    Article  Google Scholar 

  9. Berkels, B., Kondermann, C., Garbe, C.S., Rumpf, M.: Reconstructing optical flow fields by motion inpainting. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), pp. 388–400. Bonn, Germany (2009)

  10. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)

  11. Bigun, J., Granlund, G.H., Wiklund, J.: Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans. Pattern Anal. Mach. Intell. 13(8), 775–790 (1991)

    Article  Google Scholar 

  12. Black, M., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)

    Article  Google Scholar 

  13. Black, M.J., Anandan, P.: A framework for the robust estimation of optical flow. In: International Conference on Computer Vision (ICCV), pp. 231–236 (1993)

  14. Black, M.J., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. J. Comput. Vis. 25(1), 23–48 (1997)

    Article  Google Scholar 

  15. Bouguet, J.Y.: Pyramidal implementation of the affine lucas-kanade feature tracker description of the algorithm. Intel Corp. 5, 1–10 (2001)

    Google Scholar 

  16. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)

    Article  MATH  Google Scholar 

  17. Braux-Zin, J., Dupont, R., Bartoli, A.: A general dense image matching framework combining direct and feature-based costs. In: International Conference on Computer Vision (ICCV) (2013)

  18. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: European Conference on Computer Vision (ECCV), pp. 25–36. Prague, Czech Republic (2004)

  19. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)

    Article  Google Scholar 

  20. Bruhn, A., Weickert, J., Schnörr, C.: Lucas/kanade meets horn/schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)

    Article  Google Scholar 

  21. Bruhn, A., Weickert, W.: A confidence measure for variational optic flow methods. In: Geometric Properties for Incomplete Data, pp. 283–298 (2006)

  22. Bugeau, A., Ta, V., Papadakis, N.: Variational exemplar-based image colorization. IEEE Trans. Image Process. 23(1), 298–307 (2014)

    Article  MathSciNet  Google Scholar 

  23. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: European Conference on Computer Vision (ECCV), pp. 611–625. Springer-Verlag, Berlin (2012)

  24. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  25. Chan, T.F., Kang, S.H., Shen, J.: Euler’s elastica and curvature-based inpainting. SIAM J. Appl. Math. 63, 564–592 (2002)

    MathSciNet  MATH  Google Scholar 

  26. Chen, Y., Ye, X.: Projection Onto a Simplex. Cornell University Press, Ithaca (2011)

    Google Scholar 

  27. Chen, Z., Jin, H., Lin, Z., Cohen, S., Wu, Y.: Large displacement optical flow from nearest neighbor fields. In: Computer Vision and Pattern Recognition (CVPR), pp. 2443–2450 (2013)

  28. Chen, Z., Wang, J., Wu, Y.: Decomposing and regularizing sparse/non-sparse components for motion field estimation. In: Computer Vision and Pattern Recognition (CVPR), pp. 1776–1783 (2012)

  29. Corpetti, T., Mémin, E.: Stochastic uncertainty models for the luminance consistency assumption. IEEE Trans. Image Process. 21(2), 481–493 (2012)

    Article  MathSciNet  Google Scholar 

  30. Cremers, D., Soatto, S.: Motion competition: a variational approach to piecewise parametric motion segmentation. Int. J. Comput. Vis. 62(3), 249–265 (2005)

    Article  Google Scholar 

  31. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)

    Article  Google Scholar 

  32. Dong, W., Shi, G., Hu, X., Ma, Y.: Nonlocal sparse and low-rank regularization for optical flow estimation. IEEE Trans. Image Process. 23(10), 4527–4538 (2014)

    Article  MathSciNet  Google Scholar 

  33. Enkelmann, W.: Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences. Comput. Vis. Graph. Image Process. 43(2), 150–177 (1988)

    Article  Google Scholar 

  34. Fermüller, C., Shulman, D., Aloimonos, Y.: The statistics of optical flow. Comput. Vis. Image Underst. 82(1), 1–32 (2001)

    Article  MATH  Google Scholar 

  35. Fleet, D.J., Black, M.J., Yacoob, Y., Jepson, A.D.: Design and use of linear models for image motion analysis. Int. J. Comput. Vis. 36(3), 171–193 (2000)

    Article  Google Scholar 

  36. Fortun, D., Bouthemy, P., Kervrann, C.: Optical flow modeling and computation: a survey. Comput. Vis. Image Underst. 134, 1–21 (2015a)

    Article  MATH  Google Scholar 

  37. Fortun, D., Bouthemy, P., Kervrann, C.: Sparse aggregation framework for optical flow estimation. In: International Conference on Scale Space and Variational Methods in Computer Vision, pp. 323–334. Lège-Cap Ferret, France (2015b)

  38. Fortun, D., Bouthemy, P., Kervrann, C.: Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow. Comput. Vis. Image Underst. (in press) (2016)

  39. Fortun, D., Bouthemy, P., Paul-Gilloteaux, P., Kervrann, C.: Aggregation of patch-based estimations for illumination-invariant optical flow in live cell imaging. In: International Symposium on Biomedical Imaging (ISBI), pp. 660–663 (2013)

  40. Galvin, B., McCane, B., Novins, K., Mason, D., Mills, S.: Recovering motion fields: an evaluation of eight optical flow algorithms. In: British Machine Vision Conference (1998)

  41. Hafner, D., Demetz, O., Weickert, J.: Why is the census transform good for robust optic flow computation? In: Scale Space and Variational Methods in Computer Vision (SSVM), pp. 210–221 (2013)

  42. He, K., Sun, J.: Image completion approaches using the statistics of similar patches. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2423–2435 (2014)

    Article  MathSciNet  Google Scholar 

  43. Heitz, F., Bouthemy, P.: Multimodal estimation of discontinuous optical flow using markov random fields. IEEE Trans. Pattern Anal. Mach. Intell. 15(12), 1217–1232 (1993)

    Article  Google Scholar 

  44. Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)

    Article  Google Scholar 

  45. Hornacek, M., Besse, F., Kautz, J., Fitzgibbon, A.W., Rother, C.: Highly overparameterized optical flow using patchmatch belief propagation. In: European Conference on Computer Vision, Zurich, pp. 220–234 (2014)

  46. Humayun, A., Mac Aodha, O., Brostow, G.J.: Learning to find occlusion regions. In: Computer Vision and Pattern Recognition (CVPR), pp. 2161–2168 (2011)

  47. Ince, S., Konrad, J.: Occlusion-aware optical flow estimation. IEEE Trans. Image Process. 17(8), 1443–1451 (2008)

    Article  MathSciNet  Google Scholar 

  48. Jia, K., Wang, X., Tang, X.: Optical flow estimation using learned sparse model. In: International Conference on Computer Vision (ICCV), pp. 2391–2398 (2011)

  49. Jodoin, P.M., Mignotte, M.: Optical-flow based on an edge-avoidance procedure. Comput. Vis. Image Underst. 113(4), 511–531 (2009)

    Article  Google Scholar 

  50. Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans. Image Process. 16(11), 2649–2661 (2007)

    Article  MathSciNet  Google Scholar 

  51. Kondermann, C., Mester, R., Garbe, C.: A statistical confidence measure for optical flows. In: European Conference on Computer Vision (ECCV), pp. 290–301. Marseille, France (2008)

  52. Kybic, J., Nieuwenhuis, C.: Bootstrap optical flow confidence and uncertainty measure. Comput. Vis. Image Underst. 115(10), 1449–1462 (2011)

    Article  Google Scholar 

  53. Leordeanu, M., Zanfir, A., Sminchisescu, C.: Locally affine sparse-to-dense matching for motion and occlusion estimation. In: International Conference on Computer Vision (ICCV), pp. 1221–1728. Sydney, Australia (2013)

  54. Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679 (1981)

  55. Maurizot, M., Bouthemy, P., Delyon, B., Juditski, A., Odobez, J.M.: Determination of singular points in 2D deformable flow fields. In: International Conference on Image Processing (ICIP), vol. 3, pp. 488–491. Washington, DC (1995)

  56. Mémin, E., Pérez, P.: Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Trans. Image Process. 7(5), 703–719 (1998)

    Article  Google Scholar 

  57. Menze, M., Heipke, C., Geiger, A.: Discrete optimization for optical flow. In: Pattern Recognition, pp. 16–28. Springer (2015)

  58. Mohamed, M., Rashwan, H., Mertsching, B., Garcia, M., Puig, D.: Illumination-robust optical flow approach using local directional pattern. IEEE Trans. Circuits Syst. Video Technol. 24(9), 1499–1508 (2014)

    Article  Google Scholar 

  59. Mota, C., Stuke, L., Barth, E.: Analytic solutions for multiple motions. In: International Conference on Image Processing (ICIP), pp. 917–920. Thessaloniki, Greece (2001)

  60. Mozerov, M.: Constrained optical flow estimation as a matching problem. IEEE Trans. Image Process. 22(5), 2044–2055 (2013)

    Article  MathSciNet  Google Scholar 

  61. Nagel, H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8(5), 565–593 (1986)

    Article  Google Scholar 

  62. Nieuwenhuis, C., Kondermann, D., Garbe, C.S.: Complex motion models for simple optical flow estimation. In: Pattern Recognition, pp. 141–150. Springer, Berlin (2010)

  63. Odobez, J., Bouthemy, P.: Robust multiresolution estimation of parametric motion models. J. Vis. Commun. Image Represent. 6(4), 348–365 (1995)

    Article  Google Scholar 

  64. Papadakis, N., Yildizoglu, R., Aujol, J.F., Caselles, V.: High-dimension multilabel problems: convex or nonconvex relaxation? SIAM J. Imaging Sci. 6(4), 2603–2639 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  65. Pierre, F., Aujol, J.F., Bugeau, A., Papadakis, N., Ta, V.T.: Luminance-chrominance model for image colorization. SIAM J. Imaging Sci. 8(1), 536–563 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  66. Pierre, F., Aujol, J.F., Bugeau, A., Ta, V.T.: Hue constrained image colorization in the rgb space. Preprint (2014)

  67. Ranftl, R., Bredies, K., Pock, T.: Non-local total generalized variation of optical flow estimation. In: European Conference on Computer Vision, pp. 439–454. Zurich (2015)

  68. Revaud, J., Weinzaepfel, P., Harchoui Z. Schmid, C.: Epicflow: edge-preserving interpolation of correspondences for optical flow. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). Boston, MA (2015)

  69. Salmon, J., Strozecki, Y.: Patch reprojections for non-local methods. Signal Process. 92(2), 477–489 (2012)

    Article  Google Scholar 

  70. Senst, T., Eiselen, V., Sikora, T.: Robust local optical flow for feature tracking. IEEE Trans. Circuits Syst. Video Technol. 22(9), 1377–1387 (2012)

  71. Shen, X., Wu, Y.: Sparsity model for robust optical flow estimation at motion discontinuities. In: Computer Vision and Pattern Recognition (CVPR), pp. 2456–2463 (2010)

  72. Simoncelli, E.P., Adelson, E.H., Heeger, D.J.: Probability distributions of optical flow. In: Computer Vision and Pattern Recognition (CVPR), pp. 310–315 (1991)

  73. Stein, A.N., Hebert, M.: Occlusion boundaries from motion: Low-level detection and mid-level reasoning. Int. J. Comput. Vis. 82(3), 325–357 (2009)

    Article  Google Scholar 

  74. Steinbrucker, F., Pock, T., Cremers, D.: Advanced data terms for variational optic flow estimation. In: Vision, Modeling, and Visualization Workshop (2009)

  75. Sun, D., Liu, C., Pfister, H.: Local layering for joint motion estimation and occlusion detection. In: Computer Vision and Pattern Recognition (CVPR), Colombus (2014)

  76. Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106(2), 115–137 (2014)

    Article  Google Scholar 

  77. Sun, D., Sudderth, E.B., Black, M.J.: Layered segmentation and optical flow estimation over time. In: Computer Vision and Pattern Recognition (CVPR), pp. 1768–1775 (2012)

  78. Sun, J., Li, Y., Kang, S.B.: Symmetric stereo matching for occlusion handling. In: IEEE Conference on Computer Vision and Pattern (CVPR’05), pp. 399–406. San Diego, CA (2005)

  79. Timofte, R., Gool, L.V.: Sparse flow: Sparse matching for small to large displacement optical flow. In: IEEE Winter Conference on Applications of Computer Vision, WACV, Waikoloa, HI, pp. 1100–1106 (2015)

  80. Unger, M., Werlberger, M., Pock, T., Bischof, H.: Joint motion estimation and segmentation of complex scenes with label costs and occlusion modeling. In: Computer Vision and Pattern Recognition (CVPR), pp. 1878–1885 (2012)

  81. Vogel, C.R., Oman, M.E.: Iterative methods for total variation denoising. SIAM J. Sci. Comput. 17(1), 227–238 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  82. Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An improved algorithm for tv-l 1 optical flow. In: Statistical and Geometrical Approaches to Visual Motion Analysis, pp. 23–45 (2009)

  83. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C., et al.: Deepflow: Large displacement optical flow with deep matching. In: International Conference on Computer Vision (ICCV), pp. 1385–1392. Sydney (2013)

  84. Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: Computer Vision and Pattern Recognition (CVPR’10), pp. 2464–2471. San-Fransisco (2010)

  85. Wulff, J., Black, M.: Efficient sparse-to-dense optical flow estimation using a learned basis and layers. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). Boston, MA (2015)

  86. Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012)

    Article  Google Scholar 

  87. Yang, J., Li, H.: Dense, accurate optical flow estimation with piecewise parametric model. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15), Boston, MA (2015)

  88. Yang, J., Zhang, Y.: Alternating direction algorithms for \(\backslash \)ell_1-problems in compressive sensing. SIAM J. Sci. Comput. 33(1), 250–278 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  89. Zimmer, H., Bruhn, A., Weickert, J.: Optic flow in harmony. Int. J. Comput. Vis. 93(3), 1–21 (2011)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

This work was realized as part of the Quaero program, funded by OSEO, French State agency for innovation. The authors acknowledge France-BioImaging infrastructure supported by the French National Research Agency (ANR-10-INBS-04-07, “Investments for the future”). They thank also the reviewers for useful comments helping improving the paper. Finally, they thank Ferreol Soulez, Martin Storath, Olivier Demetz, Simon Setzer and Joachim Weickert for inspiring discussions at different stages of this work.

Author information

Authors and Affiliations

  1. Inria - Centre de Rennes -Bretagne Atlantique, Campus Universitaire de Beaulieu, 35042, Rennes, France

    Denis Fortun, Patrick Bouthemy & Charles Kervrann

  2. Center for Biomedical Imaging - Signal Processing core (CIBM-SP), EPFL, Lausanne, Switzerland

    Denis Fortun

  3. Biomedical Imaging Group, EPFL, Lausanne, Switzerland

    Denis Fortun

Authors
  1. Denis Fortun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick Bouthemy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charles Kervrann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denis Fortun.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fortun, D., Bouthemy, P. & Kervrann, C. A Variational Aggregation Framework for Patch-Based Optical Flow Estimation. J Math Imaging Vis 56, 280–299 (2016). https://doi.org/10.1007/s10851-016-0664-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-016-0664-6

Keywords

Search

Navigation