Advertisement

International Journal of Computer Vision

, Volume 74, Issue 1, pp 33–50 | Cite as

On the Spatial Statistics of Optical Flow

  • Stefan Roth
  • Michael J. Black
Article

Abstract

We present an analysis of the spatial and temporal statistics of “natural” optical flow fields and a novel flow algorithm that exploits their spatial statistics. Training flow fields are constructed using range images of natural scenes and 3D camera motions recovered from hand-held and car-mounted video sequences. A detailed analysis of optical flow statistics in natural scenes is presented and machine learning methods are developed to learn a Markov random field model of optical flow. The prior probability of a flow field is formulated as a Field-of-Experts model that captures the spatial statistics in overlapping patches and is trained using contrastive divergence. This new optical flow prior is compared with previous robust priors and is incorporated into a recent, accurate algorithm for dense optical flow computation. Experiments with natural and synthetic sequences illustrate how the learned optical flow prior quantitatively improves flow accuracy and how it captures the rich spatial structure found in natural scene motion.

Keywords

optical flow database of flow fields spatial statistics natural scenes Markov random fields machine learning 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 2d3 Ltd. 2002. boujou. http://www.2d3.com
  2. Alvarez, L., Weickert, J., and Sánchez, J. 2000. Reliable estimation of dense optical flow fields with large displacements. Int. J. Comput. Vision, 39(1):41–56.zbMATHCrossRefGoogle Scholar
  3. Barbu, A. and Yuille, A. 2004. Motion estimation by Swendsen-Wang cuts. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 754–761.Google Scholar
  4. Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flow techniques. Int. J. Comput. Vision, 12(1):43–77.CrossRefGoogle Scholar
  5. Ben-Ari, R. and Sochen, N. 2006. A general framework and new alignment criterion for dense optical flow. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 529–536.Google Scholar
  6. Betsch, B.Y., Einhäuser, W., Körding, K.P., and König, P. 2004. The world from a cat’s perspective—Statistics of natural videos. Biological Cybernetics, 90(1):41–50.zbMATHCrossRefGoogle Scholar
  7. Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), pp. 296–302.Google Scholar
  8. Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comput. Vis. Image Und., 63(1):75–104.CrossRefGoogle Scholar
  9. Bruhn, A. 2006. Personal Communication.Google Scholar
  10. Bruhn, A., Weickert, J., and Schnörr, C. 2005. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. Int. J. Comput. Vision, 61(3):211–231.CrossRefGoogle Scholar
  11. Calow, D., Krüger, N., Wörgötter, F., and Lappe, M. 2004. Statistics of optic flow for self-motion through natural scenes. In Dynamic Perception, Ilg, U., Bülthoff, H. and Mallot, H. (eds.), pp. 133–138.Google Scholar
  12. Cremers, D. and Soatto, S. 2005. Motion competition: A variational approach to piecewise parametric motion segmentation. Int. J. Comput. Vision, 62(3):249–265.CrossRefGoogle Scholar
  13. Davis, T.A. 2004.A column pre-ordering strategy for the unsymmetric-pattern multifrontal method. ACM Transactions on Mathematical Software, 30(2):165–195.zbMATHCrossRefGoogle Scholar
  14. Fablet, R. and Bouthemy, P. 2001. Non parametric motion recognition using temporal multiscale Gibbs models. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 501–508.Google Scholar
  15. Fermüller, C., Shulman, D., and Aloimonos, Y. 2001. The statistics of optical flow. Comput. Vis. Image Und., 82(1):1–32.zbMATHCrossRefGoogle Scholar
  16. Fleet, D.J., Black, M.J., Yacoob, Y., and Jepson, A.D. 2000. Design and use of linear models for image motion analysis. Int. J. Comput. Vision, 36(3):171–193.CrossRefGoogle Scholar
  17. Fleet, D.J., Black, M.J., and Nestares, O. 2002. Bayesian inference of visual motion boundaries. In Exploring Artificial Intelligence in the New Millennium, G. Lakemeyer and B. Nebel (eds.), Morgan Kaufmann Publisher, pp. 139–174.Google Scholar
  18. Grenander, U. and Srivastava, A. 2001. Probability models for clutter in natural images. IEEE Trans. Pattern Anal. Mach. Intell., 23(4):424–429.CrossRefGoogle Scholar
  19. Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Trans. Pattern Anal. Mach. Intell., 15(12):1217–1232.CrossRefGoogle Scholar
  20. Hinton, G.E. 1999. Products of experts. In Int. Conf. on Art. Neur. Netw. (ICANN), vol. 1, pp. 1–6.Google Scholar
  21. Hinton, G.E. 2002. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771–1800.zbMATHCrossRefGoogle Scholar
  22. Horn, B.K.P. and Schunck, B.G. 1981. Determining optical flow. Artificial Intelligence, 17(1–3):185–203.CrossRefGoogle Scholar
  23. Huang, J., Lee, A.B., and Mumford, D. 2000. Statistics of range images. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 1324ff.Google Scholar
  24. Huang, J. 2000. Statistics of Natural Images and Models. PhD thesis, Brown University.Google Scholar
  25. Irani, M. 1999. Multi-frame optical flow estimation using subspace constraints. In IEEE Int. Conf. on Comp. Vis. (ICCV), vol. 1, pp. 626–633.Google Scholar
  26. Kailath, T. 1967. The divergence and Bhattacharyya distance measures in signal selection.IEEE Transactions on Communication Technology, COM-15(1):52–60.CrossRefGoogle Scholar
  27. Konrad, J. and Dubois, E. 1988. Multigrid Bayesian estimation of image motion fields using stochastic relaxation. In IEEE Int. Conf. on Comp. Vis. (ICCV), pp. 354–362.Google Scholar
  28. Krajsek, K. and Mester, R. 2006. On the equivalence of variational and statistical differential motion estimation. In Southwest Symposium on Image Analysis and Interpretation, Denver, Colorado, pp. 11–15.Google Scholar
  29. Lee, A.B. and Huang, J. 2000. Brown range image database. http://www.dam.brown.edu/ptg/brid/index.html
  30. Lee, A.B., Mumford, D., and Huang, J. 2001. Occlusion models for natural images: A statistical study of a scale-invariant dead leaves model. Int. J. Comput. Vision, 41(1–2):35–59.zbMATHCrossRefGoogle Scholar
  31. Lewen, G.D., Bialek, W., and de Ruyter van Steveninck, R.R. 2001. Neural coding of naturalistic motion stimuli. Network: Comp. Neural, 12(3):317–329.CrossRefGoogle Scholar
  32. Lu, H. and Yuille, A.L. 2006. Ideal observers for detecting motion: Correspondence noise. In Adv. in Neur. Inf. Proc. Sys. (NIPS), vol. 18, pp. 827–834.Google Scholar
  33. Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Int. J. Conf. on Art. Intel. (IJCAI), pp. 674–679.Google Scholar
  34. Marroquin, J., Mitter, S., and Poggio, T. 1987. Probabilistic solutions of ill-posed problems in computational vision. J. Am. Stat. Assoc., 82(397):76–89.zbMATHCrossRefGoogle Scholar
  35. Martin, D., Fowlkes, C., Tal, D., and Malik, J. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE Int. Conf. on Comp. Vis. (ICCV), vol. 2, pp. 416–423.Google Scholar
  36. Mémin, É. and Pérez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. Int. J. Comput. Vision, 46(2):129–155.zbMATHCrossRefGoogle Scholar
  37. Murray, D.W. and Buxton, B.F. 1987. Scene segmentation from visual motion using global optimization. IEEE Trans. Pattern Anal. Mach. Intell., 9(2):220–228.CrossRefGoogle Scholar
  38. Olshausen, B.A. and Field, D.J. 1996. Natural image statistics and efficient coding. Network: Comp. Neural, 7(2):333–339.CrossRefGoogle Scholar
  39. Papenberg, N., Bruhn, A., Brox, T., Didas, S., and Weickert, J. 2006. Highly accurate optic flow computation with theoretically justified warping. Int. J. Comput. Vision, 67(2):141–158.CrossRefGoogle Scholar
  40. Proesmans, M., Van Gool, L.J., Pauwels, E.J., and Oosterlinck, A. 1994. Determination of optical flow and its discontinuities using non-linear diffusion. In Eur. Conf. on Comp. Vis. (ECCV), J.-O. Eklundh (ed.), vol. 801 of Lect. Notes in Comp. Sci., pp. 295–304.Google Scholar
  41. Ross, M.G. and Kaelbling, L.P. 2005. Learning static object segmentation from motion segmentation. In Nat. Conf. on Art. Int. (AAAI), Menlo Park, California, AAAI Press, pp. 956–961.Google Scholar
  42. Roth, S. and Black, M.J. 2005a. Fields of experts: A framework for learning image priors. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 2, pp. 860–867.Google Scholar
  43. Roth, S. and Black, M.J. 2005b. On the spatial statistics of optical flow. In IEEE Int. Conf. on Comp. Vis. (ICCV), vol. 1, pp. 42–49.Google Scholar
  44. Ruderman, D.L. 1994. The statistics of natural images. Network: Comp. Neural, 5(4):517–548.zbMATHCrossRefGoogle Scholar
  45. Scharr, H. 2004. Optimal filters for extended optical flow. In First International Workshop on Complex Motion, vol. 3417 of Lect. Notes in Comp. Sci., Springer.Google Scholar
  46. Scharr, H. and Spies, H. 2005. Accurate optical flow in noisy image sequences using flow adapted anisotropic diffusion. Signal Processing: Image Communication, 20(6):537–553.CrossRefGoogle Scholar
  47. Simoncelli, E.P., Adelson, E.H., and Heeger, D.J. 1991. Probability distributions of optical flow. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), pp. 310–315.Google Scholar
  48. Srivastava, A., Lee, A.B., Simoncelli, E.P., and Zhu, S.-C. 2003. On advances in statistical modeling of natural images. J. Math. Imaging Vision, 18(1):17–33.zbMATHCrossRefMathSciNetGoogle Scholar
  49. Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., and Rother, C. 2006. A comparative study of energy minimization methods for Markov random fields. In Eur. Conf. on Comp. Vis. (ECCV), A. Leonardis, H. Bischof, and A. Prinz (eds.), vol. 3952 of Lect. Notes in Comp. Sci., pp. 16–29.Google Scholar
  50. Teh, Y.W., Welling, M., Osindero, S., and Hinton, G.E. 2003. Energy-based models for sparse overcomplete representations. J. Mach. Learn. Res., 4(Dec.):1235–1260.CrossRefMathSciNetGoogle Scholar
  51. Torralba, A. 2003. Contextual priming for object detection. Int. J. Comput. Vision, 53(2):169–191.CrossRefGoogle Scholar
  52. Torralba, A. and Oliva, A. 2003. Statistics of natural image categories. Network: Comp. Neural, 14(2):391–412.CrossRefGoogle Scholar
  53. van Harteren, J.H. and Ruderman, D.L. 1998. Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. J. Roy. Stat. Soc. B, 265(1412):2315–2320.Google Scholar
  54. Weickert, J. and Schnörr, C. 2001. Variational optic flow computation with a spatio-temporal smoothness constraint. J. Math. Imaging Vision, 14(3):245–255.zbMATHCrossRefGoogle Scholar
  55. Weiss, Y. and Adelson, E.H. 1998. Slow and smooth: A Bayesian theory for the combination of local motion signals in human vision. Technical Report AI Memo 1624, MIT AI Lab, Cambridge, Massachusetts.Google Scholar
  56. Zhu, S.C. and Mumford, D. 1997. Prior learning and Gibbs reaction-diffusion. IEEE Trans. Pattern Anal. Mach. Intell., 19(11):1236–1250.CrossRefGoogle Scholar
  57. Zhu, S.C., Wu, Y., and Mumford, D. 1998. Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling. Int. J. Comput. Vision, 27(2):107–126.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Department of Computer ScienceBrown UniversityProvidenceUSA

Personalised recommendations