Advertisement

International Journal of Computer Vision

, Volume 93, Issue 3, pp 368–388 | Cite as

Optic Flow in Harmony

  • Henning Zimmer
  • Andrés Bruhn
  • Joachim Weickert
Article

Abstract

Most variational optic flow approaches just consist of three constituents: a data term, a smoothness term and a smoothness weight. In this paper, we present an approach that harmonises these three components. We start by developing an advanced data term that is robust under outliers and varying illumination conditions. This is achieved by using constraint normalisation, and an HSV colour representation with higher order constancy assumptions and a separate robust penalisation. Our novel anisotropic smoothness is designed to work complementary to the data term. To this end, it incorporates directional information from the data constraints to enable a filling-in of information solely in the direction where the data term gives no information, yielding an optimal complementary smoothing behaviour. This strategy is applied in the spatial as well as in the spatio-temporal domain. Finally, we propose a simple method for automatically determining the optimal smoothness weight. This method bases on a novel concept that we call “optimal prediction principle” (OPP). It states that the flow field obtained with the optimal smoothness weight allows for the best prediction of the next frames in the image sequence. The benefits of our “optic flow in harmony” (OFH) approach are demonstrated by an extensive experimental validation and by a competitive performance at the widely used Middlebury optic flow benchmark.

Keywords

Optic flow Variational methods Anisotropic smoothing Robust penalisation Motion tensor Parameter selection 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alvarez, L., Esclarín, J., Lefébure, M., & Sánchez, J. (1999). A PDE model for computing the optical flow. In Proc. XVI congreso de ecuaciones diferenciales y aplicaciones (pp. 1349–1356). Las Palmas de Gran Canaria, Spain. Google Scholar
  2. Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2009). A database and evaluation methodology for optical flow. Tech. Rep. MSR-TR-2009-179, Microsoft Research, Redmond, WA. Google Scholar
  3. Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994). Performance of optical flow techniques. International Journal of Computer Vision, 12(1), 43–77. CrossRefGoogle Scholar
  4. Bertero, M., Poggio, TA, & Torre, V. (1988). Ill-posed problems in early vision. Proceedings of the IEEE, 76(8), 869–889. CrossRefGoogle Scholar
  5. Bigün, J., Granlund, G. H., & Wiklund, J. (1991). Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(8), 775–790. CrossRefGoogle Scholar
  6. Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104. CrossRefGoogle Scholar
  7. Blake, A., & Zisserman, A. (1987). Visual reconstruction. Cambridge: MIT Press. Google Scholar
  8. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In T. Pajdla & J. Matas (Eds.), Computer vision—ECCV 2004, Part IV. Lecture notes in computer science (Vol. 3024, pp. 25–36). Berlin: Springer. CrossRefGoogle Scholar
  9. Bruhn, A., & Weickert, J. (2005). Towards ultimate motion estimation: combining highest accuracy with real-time performance. In Proc. tenth international conference on computer vision (Vol. 1, pp. 749–755). Beijing: IEEE Computer Society Press. Google Scholar
  10. Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. International Journal of Computer Vision, 61(3), 211–231. CrossRefGoogle Scholar
  11. Bruhn, A., Weickert, J., Kohlberger, T., & Schnörr, C. (2006). A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. International Journal of Computer Vision, 70(3), 257–277. CrossRefGoogle Scholar
  12. Charbonnier, P., Blanc-Féraud, L., Aubert, G., & Barlaud, M. (1994). Two deterministic half-quadratic regularization algorithms for computed imaging. In Proc. IEEE international conference on image processing (Vol. 2, pp. 168–172). Austin: IEEE Computer Society Press. CrossRefGoogle Scholar
  13. Cohen, I. (1993). Nonlinear variational method for optical flow computation. In Proc. eighth Scandinavian conference on image analysis (Vol. 1, pp. 523–530). Norway, Tromsø. Google Scholar
  14. Coons, S. A. (1967). Surfaces for computer aided design of space forms (Tech. Rep. MIT/LCS/TR-41). Massachusetts Institute of Technology, Cambridge. Google Scholar
  15. Elsgolc, L. E. (1962). Calculus of variations. London: Pergamon. Google Scholar
  16. Fleet, D. J., & Jepson, A. D. (1990). Computation of component image velocity from local phase information. International Journal of Computer Vision, 5(1), 77–104. CrossRefGoogle Scholar
  17. Förstner, W., & Gülch, E. (1987). A fast operator for detection and precise location of distinct points, corners and centres of circular features. In Proc. ISPRS intercommission conference on fast processing of photogrammetric data (pp. 281–305). Interlaken, Switzerland. Google Scholar
  18. Golland, P., & Bruckstein, A. M. (1997). Motion from color. Computer Vision and Image Understanding, 68(3), 346–362. CrossRefGoogle Scholar
  19. Grossauer, H., & Thoman, P. (2008). GPU-based multigrid: real-time performance in high resolution nonlinear image processing. In A. Gasteratos, M. Vincze, & J. K. Tsotsos (Eds.), Lecture notes in computer science: Vol. 5008. Computer vision systems (pp. 141–150). Berlin: Springer. CrossRefGoogle Scholar
  20. Gwosdek, P., Zimmer, H., Grewenig, S., Bruhn, A., & Weickert, J. (2010). A highly efficient GPU implementation for variational optic flow based on the Euler-Lagrange framework. In Proc. 2010 ECCV workshop on computer vision with GPUs, Heraklion, Greece. Google Scholar
  21. Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203. CrossRefGoogle Scholar
  22. Krajsek, K., & Mester, R. (2007). Bayesian model selection for optical flow estimation. In F. A. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Pattern recognition. Lecture notes in computer science (pp. 142–151). Berlin: Springer. CrossRefGoogle Scholar
  23. Lai, S. H., & Vemuri, B. C. (1998). Reliable and efficient computation of optical flow. International Journal of Computer Vision, 29(2), 87–105. CrossRefGoogle Scholar
  24. Lei, C., & Yang, Y. H. (2009). Optical flow estimation on coarse-to-fine region-trees using discrete optimization. In Proc. 2009 IEEE international conference on computer vision. Kyoto: IEEE Computer Society Press. Google Scholar
  25. Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In Proc. seventh international joint conference on artificial intelligence (pp. 674–679). Vancouver, Canada. Google Scholar
  26. Mileva, Y., Bruhn, A., & Weickert, J. (2007). Illumination-robust variational optical flow with photometric invariants. In F. A. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Pattern recognition. Lecture notes in computer science (pp. 152–162). Berlin: Springer. CrossRefGoogle Scholar
  27. Murray, D. W., & Buxton, B. F. (1987). Scene segmentation from visual motion using global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2), 220–228. CrossRefGoogle Scholar
  28. Nagel, H. H. (1990). Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In O. Faugeras (Ed.), Lecture notes in computer science: Vol. 427. Computer vision—ECCV ’90 (pp. 139–148). Berlin: Springer. Google Scholar
  29. Nagel, H. H., & Enkelmann, W. (1986). An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 565–593. CrossRefGoogle Scholar
  30. Ng, L., & Solo, V. (1997). A data-driven method for choosing smoothing parameters in optical flow problems. In Proc. 1997 IEEE international conference on image processing (Vol. 3, pp. 360–363). Los Alamitos: IEEE Computer Society. CrossRefGoogle Scholar
  31. Perona, P., & Malik, J. (1990). Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 629–639. CrossRefGoogle Scholar
  32. Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268. CrossRefMATHGoogle Scholar
  33. Schnörr, C. (1993). On functionals with greyvalue-controlled smoothness terms for determining optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(10), 1074–1079. CrossRefGoogle Scholar
  34. Schnörr, C. (1994). Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. twelfth international conference on pattern recognition (Vol. A, pp. 661–663). Jerusalem: IEEE Computer Society Press. Google Scholar
  35. Schoenemann, T., & Cremers, D. (2006). Near real-time motion segmentation using graph cuts. In K. Franke, K. R. Müller, B. Nickolay, & R. Schäfer (Eds.), Pattern recognition, Lecture notes in computer science (pp. 455–464). Berlin: Springer. CrossRefGoogle Scholar
  36. Shulman, D., & Hervé, J. (1989). Regularization of discontinuous flow fields. In Proc. workshop on visual motion (pp. 81–86). Irvine: IEEE Computer Society Press. CrossRefGoogle Scholar
  37. Simoncelli, E. P., Adelson, E. H., & Heeger, D. J. (1991). Probability distributions of optical flow. In Proc. 1991 IEEE computer society conference on computer vision and pattern recognition (pp. 310–315). Maui: IEEE Computer Society Press. CrossRefGoogle Scholar
  38. Sun, D., Roth, S., Lewis, J. P., & Black, M. J. (2008). Learning optical flow. In D. Forsyth, P. Torr, & A. Zisserman (Eds.), Computer vision—ECCV 2008, Part III, Lecture notes in computer science (pp. 83–97). Berlin: Springer. CrossRefGoogle Scholar
  39. Sun, D., Roth, S., & Black, M. J. (2010). Secrets of optical flow estimation and their principles. In Proc. 2010 IEEE computer society conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society Press. Google Scholar
  40. Sundaram, N., Brox, T., & Keutzer, K. (2010). Dense point trajectories by GPU-accelerated large displacement optical flow. In Lecture notes in computer science: Vol. 6311. Computer vision—ECCV 2010 (pp. 438–451). Berlin: Springer. CrossRefGoogle Scholar
  41. Tretiak, O., & Pastor, L. (1984). Velocity estimation from image sequences with second order differential operators. In Proc. seventh international conference on pattern recognition (pp. 16–19). Montreal, Canada. Google Scholar
  42. van de Weijer, J., & Gevers, T. (2004). Robust optical flow from photometric invariants. In Proc. 2004 IEEE international conference on image processing (Vol. 3, pp. 1835–1838). Singapore: IEEE Signal Processing Society. Google Scholar
  43. Wedel, A., Pock, T., Zach, C., Bischof, H., & Cremers, D. (2008). An improved algorithm for TV-L 1 optical flow computation. In D. Cremers, B. Rosenhahn, A. L. Yuille, & F. R. Schmidt (Eds.), Lecture notes in computer science: Vol. 5604. Statistical and geometrical approaches to visual motion analysis (pp. 23–45). Berlin: Springer. Google Scholar
  44. Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In Proc. 2009 IEEE international conference on computer vision. Kyoto: IEEE Computer Society Press. Google Scholar
  45. Weickert, J. (1996). Theoretical foundations of anisotropic diffusion in image processing. Computing Supplement, 11, 221–236. Google Scholar
  46. Weickert, J., & Schnörr, C. (2001a). A theoretical framework for convex regularizers in PDE-based computation of image motion. International Journal of Computer Vision, 45(3), 245–264. CrossRefMATHGoogle Scholar
  47. Weickert, J., & Schnörr, C. (2001b). Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3), 245–255. CrossRefMATHGoogle Scholar
  48. Werlberger, M., Pock, T., & Bischof, H. (2010). Motion estimation with non-local total variation regularization. In Proc. 2010 IEEE computer society conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society Press. Google Scholar
  49. Xiao, J., Cheng, H., Sawhney, H., Rao, C., & Isnardi, M. (2006). Bilateral filtering-based optical flow estimation with occlusion detection. In A. Leonardis, H. Bischof, & A. Pinz (Eds.), Lecture notes in computer science: Vol. 3951. Computer vision—ECCV 2006, Part I (pp. 211–224). Berlin: Springer. Google Scholar
  50. Xu, L., Jia, J., & Matsushita, Y. (2010). Motion detail preserving optical flow estimation. In Proc. 2010 IEEE computer society conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society Press. Google Scholar
  51. Yaroslavsky, L. P. (1985). Digital picture processing: an introduction. Berlin: Springer. MATHGoogle Scholar
  52. Yoon, K. J., & Kweon, I. S. (2006). Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 650–656. CrossRefGoogle Scholar
  53. Zach, C., Pock, T., & Bischof, H. (2007). A duality based approach for realtime TV-L 1 optical flow. In F. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Lecture notes in computer science: Vol. 4713. Pattern recognition (pp. 214–223). Berlin: Springer. CrossRefGoogle Scholar
  54. Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H. P. (2009). Complementary optic flow. In D. Cremers, Y. Boykov, A. Blake, & F. R. Schmidt (Eds.), Lecture notes in computer science: Vol. 5681. Energy minimization methods in computer vision and pattern recognition (EMMCVPR) (pp. 207–220). Berlin: Springer. CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Henning Zimmer
    • 1
  • Andrés Bruhn
    • 2
  • Joachim Weickert
    • 1
  1. 1.Mathematical Image Analysis Group, Faculty of Mathematics and Computer ScienceSaarland UniversitySaarbrückenGermany
  2. 2.Vision and Image Processing Group, Cluster of Excellence Multimodal Computing and InteractionSaarland UniversitySaarbrückenGermany

Personalised recommendations