Advertisement

Journal of Mathematical Imaging and Vision

, Volume 56, Issue 2, pp 280–299 | Cite as

A Variational Aggregation Framework for Patch-Based Optical Flow Estimation

  • Denis FortunEmail author
  • Patrick Bouthemy
  • Charles Kervrann
Article

Abstract

We propose a variational aggregation method for optical flow estimation. It consists of a two-step framework, first estimating a collection of parametric motion models to generate motion candidates, and then reconstructing a global dense motion field. The aggregation step is designed as a motion reconstruction problem from spatially varying sets of motion candidates given by parametric motion models. Our method is designed to capture large displacements in a variational framework without requiring any coarse-to-fine strategy. We handle occlusion with a motion inpainting approach in the candidates computation step. By performing parametric motion estimation, we combine the robustness to noise of local parametric methods with the accuracy yielded by global regularization. We demonstrate the performance of our aggregation approach by comparing it to standard variational methods and a discrete aggregation approach on the Middlebury and MPI Sintel datasets.

Keywords

Optical flow Parametric motion Aggregation Variational optimization 

Notes

Acknowledgments

This work was realized as part of the Quaero program, funded by OSEO, French State agency for innovation. The authors acknowledge France-BioImaging infrastructure supported by the French National Research Agency (ANR-10-INBS-04-07, “Investments for the future”). They thank also the reviewers for useful comments helping improving the paper. Finally, they thank Ferreol Soulez, Martin Storath, Olivier Demetz, Simon Setzer and Joachim Weickert for inspiring discussions at different stages of this work.

References

  1. 1.
    Alba, A., Arce-Santana, E., Riviera, M.: Optical flow estimation with prior models obtained from phase correlation. In: Advances in Visual Computing, pp. 417–426. Springer, Berlin (2010)Google Scholar
  2. 2.
    Arias, P., Facciolo, G., Caselles, V., Sapiro, G.: A variational framework for exemplar-based image inpainting. Int. J. Comput. Vis. 93(3), 319–347 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Ayvaci, A., Raptis, M., Soatto, S.: Sparse occlusion detection with optical flow. Int. J. Comput. Vis. 97(3), 322–338 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Bailer, C., Taetz, B., Stricker, D.: Flow fields: dense correspondence fields for highly accurate large displacement optical flow estimation. In: IEEE International Conference on Computer Vision, pp. 4015–4023 (2015)Google Scholar
  5. 5.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92(1), 1–31 (2011)CrossRefGoogle Scholar
  6. 6.
    Bao, L., Yang, Q., Jin, H.: Fast edge-preserving patchmatch for large displacement optical flow. In: Computer Vision and Pattern Recognition (CVPR), Columbus (2014)Google Scholar
  7. 7.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24 (2009)CrossRefGoogle Scholar
  8. 8.
    Barron, J., Fleet, D., Beauchemin, S.: Evaluation of optical flow. Int. J. Comput. Vis. 12(1), 43–77 (1994)CrossRefGoogle Scholar
  9. 9.
    Berkels, B., Kondermann, C., Garbe, C.S., Rumpf, M.: Reconstructing optical flow fields by motion inpainting. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), pp. 388–400. Bonn, Germany (2009)Google Scholar
  10. 10.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)Google Scholar
  11. 11.
    Bigun, J., Granlund, G.H., Wiklund, J.: Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Trans. Pattern Anal. Mach. Intell. 13(8), 775–790 (1991)CrossRefGoogle Scholar
  12. 12.
    Black, M., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63(1), 75–104 (1996)CrossRefGoogle Scholar
  13. 13.
    Black, M.J., Anandan, P.: A framework for the robust estimation of optical flow. In: International Conference on Computer Vision (ICCV), pp. 231–236 (1993)Google Scholar
  14. 14.
    Black, M.J., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. J. Comput. Vis. 25(1), 23–48 (1997)CrossRefGoogle Scholar
  15. 15.
    Bouguet, J.Y.: Pyramidal implementation of the affine lucas-kanade feature tracker description of the algorithm. Intel Corp. 5, 1–10 (2001)Google Scholar
  16. 16.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)zbMATHCrossRefGoogle Scholar
  17. 17.
    Braux-Zin, J., Dupont, R., Bartoli, A.: A general dense image matching framework combining direct and feature-based costs. In: International Conference on Computer Vision (ICCV) (2013)Google Scholar
  18. 18.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: European Conference on Computer Vision (ECCV), pp. 25–36. Prague, Czech Republic (2004)Google Scholar
  19. 19.
    Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)CrossRefGoogle Scholar
  20. 20.
    Bruhn, A., Weickert, J., Schnörr, C.: Lucas/kanade meets horn/schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)CrossRefGoogle Scholar
  21. 21.
    Bruhn, A., Weickert, W.: A confidence measure for variational optic flow methods. In: Geometric Properties for Incomplete Data, pp. 283–298 (2006)Google Scholar
  22. 22.
    Bugeau, A., Ta, V., Papadakis, N.: Variational exemplar-based image colorization. IEEE Trans. Image Process. 23(1), 298–307 (2014)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: European Conference on Computer Vision (ECCV), pp. 611–625. Springer-Verlag, Berlin (2012)Google Scholar
  24. 24.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Chan, T.F., Kang, S.H., Shen, J.: Euler’s elastica and curvature-based inpainting. SIAM J. Appl. Math. 63, 564–592 (2002)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Chen, Y., Ye, X.: Projection Onto a Simplex. Cornell University Press, Ithaca (2011)Google Scholar
  27. 27.
    Chen, Z., Jin, H., Lin, Z., Cohen, S., Wu, Y.: Large displacement optical flow from nearest neighbor fields. In: Computer Vision and Pattern Recognition (CVPR), pp. 2443–2450 (2013)Google Scholar
  28. 28.
    Chen, Z., Wang, J., Wu, Y.: Decomposing and regularizing sparse/non-sparse components for motion field estimation. In: Computer Vision and Pattern Recognition (CVPR), pp. 1776–1783 (2012)Google Scholar
  29. 29.
    Corpetti, T., Mémin, E.: Stochastic uncertainty models for the luminance consistency assumption. IEEE Trans. Image Process. 21(2), 481–493 (2012)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Cremers, D., Soatto, S.: Motion competition: a variational approach to piecewise parametric motion segmentation. Int. J. Comput. Vis. 62(3), 249–265 (2005)CrossRefGoogle Scholar
  31. 31.
    Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  32. 32.
    Dong, W., Shi, G., Hu, X., Ma, Y.: Nonlocal sparse and low-rank regularization for optical flow estimation. IEEE Trans. Image Process. 23(10), 4527–4538 (2014)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Enkelmann, W.: Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences. Comput. Vis. Graph. Image Process. 43(2), 150–177 (1988)CrossRefGoogle Scholar
  34. 34.
    Fermüller, C., Shulman, D., Aloimonos, Y.: The statistics of optical flow. Comput. Vis. Image Underst. 82(1), 1–32 (2001)zbMATHCrossRefGoogle Scholar
  35. 35.
    Fleet, D.J., Black, M.J., Yacoob, Y., Jepson, A.D.: Design and use of linear models for image motion analysis. Int. J. Comput. Vis. 36(3), 171–193 (2000)CrossRefGoogle Scholar
  36. 36.
    Fortun, D., Bouthemy, P., Kervrann, C.: Optical flow modeling and computation: a survey. Comput. Vis. Image Underst. 134, 1–21 (2015a)CrossRefzbMATHGoogle Scholar
  37. 37.
    Fortun, D., Bouthemy, P., Kervrann, C.: Sparse aggregation framework for optical flow estimation. In: International Conference on Scale Space and Variational Methods in Computer Vision, pp. 323–334. Lège-Cap Ferret, France (2015b)Google Scholar
  38. 38.
    Fortun, D., Bouthemy, P., Kervrann, C.: Aggregation of local parametric candidates with exemplar-based occlusion handling for optical flow. Comput. Vis. Image Underst. (in press) (2016)Google Scholar
  39. 39.
    Fortun, D., Bouthemy, P., Paul-Gilloteaux, P., Kervrann, C.: Aggregation of patch-based estimations for illumination-invariant optical flow in live cell imaging. In: International Symposium on Biomedical Imaging (ISBI), pp. 660–663 (2013)Google Scholar
  40. 40.
    Galvin, B., McCane, B., Novins, K., Mason, D., Mills, S.: Recovering motion fields: an evaluation of eight optical flow algorithms. In: British Machine Vision Conference (1998)Google Scholar
  41. 41.
    Hafner, D., Demetz, O., Weickert, J.: Why is the census transform good for robust optic flow computation? In: Scale Space and Variational Methods in Computer Vision (SSVM), pp. 210–221 (2013)Google Scholar
  42. 42.
    He, K., Sun, J.: Image completion approaches using the statistics of similar patches. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2423–2435 (2014)CrossRefMathSciNetGoogle Scholar
  43. 43.
    Heitz, F., Bouthemy, P.: Multimodal estimation of discontinuous optical flow using markov random fields. IEEE Trans. Pattern Anal. Mach. Intell. 15(12), 1217–1232 (1993)CrossRefGoogle Scholar
  44. 44.
    Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)CrossRefGoogle Scholar
  45. 45.
    Hornacek, M., Besse, F., Kautz, J., Fitzgibbon, A.W., Rother, C.: Highly overparameterized optical flow using patchmatch belief propagation. In: European Conference on Computer Vision, Zurich, pp. 220–234 (2014)Google Scholar
  46. 46.
    Humayun, A., Mac Aodha, O., Brostow, G.J.: Learning to find occlusion regions. In: Computer Vision and Pattern Recognition (CVPR), pp. 2161–2168 (2011)Google Scholar
  47. 47.
    Ince, S., Konrad, J.: Occlusion-aware optical flow estimation. IEEE Trans. Image Process. 17(8), 1443–1451 (2008)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Jia, K., Wang, X., Tang, X.: Optical flow estimation using learned sparse model. In: International Conference on Computer Vision (ICCV), pp. 2391–2398 (2011)Google Scholar
  49. 49.
    Jodoin, P.M., Mignotte, M.: Optical-flow based on an edge-avoidance procedure. Comput. Vis. Image Underst. 113(4), 511–531 (2009)CrossRefGoogle Scholar
  50. 50.
    Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans. Image Process. 16(11), 2649–2661 (2007)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Kondermann, C., Mester, R., Garbe, C.: A statistical confidence measure for optical flows. In: European Conference on Computer Vision (ECCV), pp. 290–301. Marseille, France (2008)Google Scholar
  52. 52.
    Kybic, J., Nieuwenhuis, C.: Bootstrap optical flow confidence and uncertainty measure. Comput. Vis. Image Underst. 115(10), 1449–1462 (2011)CrossRefGoogle Scholar
  53. 53.
    Leordeanu, M., Zanfir, A., Sminchisescu, C.: Locally affine sparse-to-dense matching for motion and occlusion estimation. In: International Conference on Computer Vision (ICCV), pp. 1221–1728. Sydney, Australia (2013)Google Scholar
  54. 54.
    Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 674–679 (1981)Google Scholar
  55. 55.
    Maurizot, M., Bouthemy, P., Delyon, B., Juditski, A., Odobez, J.M.: Determination of singular points in 2D deformable flow fields. In: International Conference on Image Processing (ICIP), vol. 3, pp. 488–491. Washington, DC (1995)Google Scholar
  56. 56.
    Mémin, E., Pérez, P.: Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Trans. Image Process. 7(5), 703–719 (1998)CrossRefGoogle Scholar
  57. 57.
    Menze, M., Heipke, C., Geiger, A.: Discrete optimization for optical flow. In: Pattern Recognition, pp. 16–28. Springer (2015)Google Scholar
  58. 58.
    Mohamed, M., Rashwan, H., Mertsching, B., Garcia, M., Puig, D.: Illumination-robust optical flow approach using local directional pattern. IEEE Trans. Circuits Syst. Video Technol. 24(9), 1499–1508 (2014)CrossRefGoogle Scholar
  59. 59.
    Mota, C., Stuke, L., Barth, E.: Analytic solutions for multiple motions. In: International Conference on Image Processing (ICIP), pp. 917–920. Thessaloniki, Greece (2001)Google Scholar
  60. 60.
    Mozerov, M.: Constrained optical flow estimation as a matching problem. IEEE Trans. Image Process. 22(5), 2044–2055 (2013)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Nagel, H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8(5), 565–593 (1986)CrossRefGoogle Scholar
  62. 62.
    Nieuwenhuis, C., Kondermann, D., Garbe, C.S.: Complex motion models for simple optical flow estimation. In: Pattern Recognition, pp. 141–150. Springer, Berlin (2010)Google Scholar
  63. 63.
    Odobez, J., Bouthemy, P.: Robust multiresolution estimation of parametric motion models. J. Vis. Commun. Image Represent. 6(4), 348–365 (1995)CrossRefGoogle Scholar
  64. 64.
    Papadakis, N., Yildizoglu, R., Aujol, J.F., Caselles, V.: High-dimension multilabel problems: convex or nonconvex relaxation? SIAM J. Imaging Sci. 6(4), 2603–2639 (2013)MathSciNetzbMATHCrossRefGoogle Scholar
  65. 65.
    Pierre, F., Aujol, J.F., Bugeau, A., Papadakis, N., Ta, V.T.: Luminance-chrominance model for image colorization. SIAM J. Imaging Sci. 8(1), 536–563 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  66. 66.
    Pierre, F., Aujol, J.F., Bugeau, A., Ta, V.T.: Hue constrained image colorization in the rgb space. Preprint (2014)Google Scholar
  67. 67.
    Ranftl, R., Bredies, K., Pock, T.: Non-local total generalized variation of optical flow estimation. In: European Conference on Computer Vision, pp. 439–454. Zurich (2015)Google Scholar
  68. 68.
    Revaud, J., Weinzaepfel, P., Harchoui Z. Schmid, C.: Epicflow: edge-preserving interpolation of correspondences for optical flow. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). Boston, MA (2015)Google Scholar
  69. 69.
    Salmon, J., Strozecki, Y.: Patch reprojections for non-local methods. Signal Process. 92(2), 477–489 (2012)CrossRefGoogle Scholar
  70. 70.
    Senst, T., Eiselen, V., Sikora, T.: Robust local optical flow for feature tracking. IEEE Trans. Circuits Syst. Video Technol. 22(9), 1377–1387 (2012)Google Scholar
  71. 71.
    Shen, X., Wu, Y.: Sparsity model for robust optical flow estimation at motion discontinuities. In: Computer Vision and Pattern Recognition (CVPR), pp. 2456–2463 (2010)Google Scholar
  72. 72.
    Simoncelli, E.P., Adelson, E.H., Heeger, D.J.: Probability distributions of optical flow. In: Computer Vision and Pattern Recognition (CVPR), pp. 310–315 (1991)Google Scholar
  73. 73.
    Stein, A.N., Hebert, M.: Occlusion boundaries from motion: Low-level detection and mid-level reasoning. Int. J. Comput. Vis. 82(3), 325–357 (2009)CrossRefGoogle Scholar
  74. 74.
    Steinbrucker, F., Pock, T., Cremers, D.: Advanced data terms for variational optic flow estimation. In: Vision, Modeling, and Visualization Workshop (2009)Google Scholar
  75. 75.
    Sun, D., Liu, C., Pfister, H.: Local layering for joint motion estimation and occlusion detection. In: Computer Vision and Pattern Recognition (CVPR), Colombus (2014)Google Scholar
  76. 76.
    Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106(2), 115–137 (2014)CrossRefGoogle Scholar
  77. 77.
    Sun, D., Sudderth, E.B., Black, M.J.: Layered segmentation and optical flow estimation over time. In: Computer Vision and Pattern Recognition (CVPR), pp. 1768–1775 (2012)Google Scholar
  78. 78.
    Sun, J., Li, Y., Kang, S.B.: Symmetric stereo matching for occlusion handling. In: IEEE Conference on Computer Vision and Pattern (CVPR’05), pp. 399–406. San Diego, CA (2005)Google Scholar
  79. 79.
    Timofte, R., Gool, L.V.: Sparse flow: Sparse matching for small to large displacement optical flow. In: IEEE Winter Conference on Applications of Computer Vision, WACV, Waikoloa, HI, pp. 1100–1106 (2015)Google Scholar
  80. 80.
    Unger, M., Werlberger, M., Pock, T., Bischof, H.: Joint motion estimation and segmentation of complex scenes with label costs and occlusion modeling. In: Computer Vision and Pattern Recognition (CVPR), pp. 1878–1885 (2012)Google Scholar
  81. 81.
    Vogel, C.R., Oman, M.E.: Iterative methods for total variation denoising. SIAM J. Sci. Comput. 17(1), 227–238 (1996)MathSciNetzbMATHCrossRefGoogle Scholar
  82. 82.
    Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An improved algorithm for tv-l 1 optical flow. In: Statistical and Geometrical Approaches to Visual Motion Analysis, pp. 23–45 (2009)Google Scholar
  83. 83.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C., et al.: Deepflow: Large displacement optical flow with deep matching. In: International Conference on Computer Vision (ICCV), pp. 1385–1392. Sydney (2013)Google Scholar
  84. 84.
    Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: Computer Vision and Pattern Recognition (CVPR’10), pp. 2464–2471. San-Fransisco (2010)Google Scholar
  85. 85.
    Wulff, J., Black, M.: Efficient sparse-to-dense optical flow estimation using a learned basis and layers. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15). Boston, MA (2015)Google Scholar
  86. 86.
    Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 34(9), 1744–1757 (2012)CrossRefGoogle Scholar
  87. 87.
    Yang, J., Li, H.: Dense, accurate optical flow estimation with piecewise parametric model. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15), Boston, MA (2015)Google Scholar
  88. 88.
    Yang, J., Zhang, Y.: Alternating direction algorithms for \(\backslash \)ell_1-problems in compressive sensing. SIAM J. Sci. Comput. 33(1), 250–278 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  89. 89.
    Zimmer, H., Bruhn, A., Weickert, J.: Optic flow in harmony. Int. J. Comput. Vis. 93(3), 1–21 (2011)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Denis Fortun
    • 1
    • 2
    • 3
    Email author
  • Patrick Bouthemy
    • 1
  • Charles Kervrann
    • 1
  1. 1.Inria - Centre de Rennes -Bretagne AtlantiqueRennesFrance
  2. 2.Center for Biomedical Imaging - Signal Processing core (CIBM-SP)EPFLLausanneSwitzerland
  3. 3.Biomedical Imaging Group, EPFLLausanneSwitzerland

Personalised recommendations