International Journal of Computer Vision

, Volume 92, Issue 1, pp 1–31 | Cite as

A Database and Evaluation Methodology for Optical Flow

  • Simon Baker
  • Daniel Scharstein
  • J. P. Lewis
  • Stefan Roth
  • Michael J. Black
  • Richard Szeliski
Open Access
Article

Abstract

The quantitative evaluation of optical flow algorithms by Barron et al. (1994) led to significant advances in performance. The challenges for optical flow algorithms today go beyond the datasets and evaluation methods proposed in that paper. Instead, they center on problems associated with complex natural scenes, including nonrigid motion, real sensor noise, and motion discontinuities. We propose a new set of benchmarks and evaluation methods for the next generation of optical flow algorithms. To that end, we contribute four types of data to test different aspects of optical flow algorithms: (1) sequences with nonrigid motion where the ground-truth flow is determined by tracking hidden fluorescent texture, (2) realistic synthetic sequences, (3) high frame-rate video used to study interpolation error, and (4) modified stereo sequences of static scenes. In addition to the average angular error used by Barron et al., we compute the absolute flow endpoint error, measures for frame interpolation error, improved statistics, and results at motion discontinuities and in textureless regions. In October 2007, we published the performance of several well-known methods on a preliminary version of our data to establish the current state of the art. We also made the data freely available on the web at http://vision.middlebury.edu/flow/. Subsequently a number of researchers have uploaded their results to our website and published papers using the data. A significant improvement in performance has already been achieved. In this paper we analyze the results obtained to date and draw a large number of conclusions from them.

Keywords

Optical flow Survey Algorithms Database Benchmarks Evaluation Metrics 

References

  1. Adiv, G. (1985). Determining three-dimensional motion and structure from optical flow generated by several moving objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 7(4), 384–401. CrossRefGoogle Scholar
  2. Aggarwal, J., & Nandhakumar, N. (1988). On the computation of motion from sequences of images—a review. Proceedings of the IEEE, 76(8), 917–935. CrossRefGoogle Scholar
  3. Anandan, P. (1989). A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2(3), 283–310. CrossRefGoogle Scholar
  4. Anandan, P., & Weiss, R. (1985). Introducing smoothness constraint in a matching approach for the computation of displacement fields. In Proceedings of the DARPA image understanding workshop (pp. 186–196). Google Scholar
  5. Baker, S., & Matthews, I. (2004). Lucas-Kanade 20 years on: a unifying framework. International Journal of Computer Vision, 46(3), 221–255. CrossRefGoogle Scholar
  6. Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., & Szeliski, R. (2007). A database and evaluation methodology for optical flow. In Proceedings of the IEEE international conference on computer vision. Google Scholar
  7. Barron, J., Fleet, D., & Beauchemin, S. (1994). Performance of optical flow techniques. International Journal of Computer Vision, 12(1), 43–77. CrossRefGoogle Scholar
  8. Battiti, R., Amaldi, E., & Koch, C. (1991). Computing optical flow across multiple scales: an adaptive coarse-to-fine strategy. International Journal of Computer Vision, 6(2), 133–145. CrossRefGoogle Scholar
  9. Beier, T., & Neely, S. (1992). Feature-based image metamorphosis. In Annual conference series: Vol. 26(2). ACM computer graphics, SIGGRAPH (pp. 35–42). Google Scholar
  10. Bergen, J., Anandan, P., Hanna, K., & Hingorani, R. (1992). Hierarchical model-based motion estimation. In Proceedings of the European conference on computer vision (pp. 237–252). Google Scholar
  11. Black, M., & Anandan, P. (1991). Robust dynamic motion estimation over time. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 296–302). CrossRefGoogle Scholar
  12. Black, M., & Anandan, P. (1996). The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104. CrossRefGoogle Scholar
  13. Black, M., & Jepson, A. (1996). Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(10), 972–986. CrossRefGoogle Scholar
  14. Blake, A., & Zisserman, A. (1987). Visual reconstruction. Cambridge: MIT Press. Google Scholar
  15. Bleyer, M., & Chambon, S. (2010). Does color really help in dense stereo matching? In Proceedings of the international symposium 3D data processing, visualization and transmission. Google Scholar
  16. Bleyer, M., Rother, C., & Kohli, P. (2010). Surface stereo with soft segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  17. Boykov, Y., Veksler, O., & Zabih, R. (2001). Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(11), 1222–1239. CrossRefGoogle Scholar
  18. Brox, T., Bregler, C., & Malik, J. (2009). Large displacement optical flow. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  19. Brox, T., Bruhn, A., Papenberg, N., &Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Proceedings of the European Conference on Computer Vision (Vol. 4, pp. 25–36). Google Scholar
  20. Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. International Journal of Computer Vision, 61(3), 211–231. CrossRefGoogle Scholar
  21. Bruhn, A., Weickert, J., Kohlberger, T., & Schnörr, C. (2006). A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. International Journal of Computer Vision, 70(3), 257–277. CrossRefGoogle Scholar
  22. Burt, P., Yen, C., & Xu, X. (1982). Local correlation measures for motion analysis: a comparative study. In Proceedings of the IEEE conference on pattern recognition and image processing (pp. 269–274). Google Scholar
  23. Burt, P., Yen, C., & Xu, X. (1983). Multi-resolution flow-through motion analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 246–252). Google Scholar
  24. Cassisa, C., Simoens, S., & Prinet, V. (2009). Two-frame optical flow formulation in an unwarped multiresolution scheme. In Proceedings of the Iberoamerican congress on pattern recognition (pp. 790–797). Google Scholar
  25. Cooke, T. (2008). Two applications of graph-cuts to image processing. In Proceedings of digital image computing: techniques and applications (pp. 498–504). CrossRefGoogle Scholar
  26. DNA Research (2008). 3Delight rendering software. http://www.3delight.com/.
  27. Enkelman, W. (1986). Investigations of multigrid algorithms for the estimation of optical flow fields in image sequences. In Proceedings of the workshop on motion: representations and analysis (pp. 81–87). Google Scholar
  28. Everingham, M., Van Gool, L., Williams, C., Winn, J., & Zisserman, A. (2009). The PASCAL visual object classes challenge 2009. http://www.pascal-network.org/challenges/VOC/voc2009/workshop/index.html
  29. Fei-Fei, L., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 594–611. CrossRefGoogle Scholar
  30. Fleet, D., & Jepson, A. (1990). Computation of component image velocity from local phase information. International Journal of Computer Vision, 5(1), 77–104. CrossRefGoogle Scholar
  31. Fuh, C., & Maragos, P. (1989). Region-based optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 130–135). Google Scholar
  32. Georghiades, A., Belhumeur, P., & Kriegman, D. (2001). From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 643–660. CrossRefGoogle Scholar
  33. Glazer, F., Reyonds, G., & Anandan, P. (1983). Scene matching by hierarchical correlation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 432–441). Google Scholar
  34. Glocker, B., Paragios, N., Komodakis, N., Tziritas, G., & Navab, N. (2008). Optical flow estimation with uncertainties through dynamic MRFs. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  35. Golland, P., & Bruckstein, A. (1997). Motion from color. Computer Vision and Image Understanding, 68(3), 346–362. CrossRefGoogle Scholar
  36. Gross, R., Matthews, I., Cohn, J., Kanade, T., & Baker, S. (2008). Multi-PIE. In Proceedings of the international conference on automatic face and gesture recognition. Google Scholar
  37. Hanna, K. (1991). Direct multi-resolution estimation of ego-motion and structure from motion. In Proceedings of the IEEE workshop on visual motion (pp. 156–162). CrossRefGoogle Scholar
  38. Haussecker, H., & Fleet, D. (2000). Computing optical flow with physical models of brightness variation. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 760–767). Google Scholar
  39. Herbst, E., Seitz, S., & Baker, S. (2009). Occlusion reasoning for temporal interpolation using optical flow. Technical report UW-CSE-09-08-01, Department of Computer Science and Engineering University of Washington. Google Scholar
  40. Horn, B. (1986). Robot vision. Cambridge: MIT Press. Google Scholar
  41. Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203. CrossRefGoogle Scholar
  42. Jepson, A., & Black, M. (1993). Mixture models for optical flow computation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 760–761). CrossRefGoogle Scholar
  43. Ju, S. (1998). Estimating image motion in layers: the skin and bones model. PhD thesis, Department of Computer Science, University of Toronto. Google Scholar
  44. Ju, S., Black, M., & Jepson, A. (1996). Skin and bones: multi-layer, locally affine, optical flow and regularization of transparency. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 307–314). Google Scholar
  45. Jung, H., Lee, K., & Lee, S. (2008). Toward global minimum through combined local minima. In Proceedings of the European conference on computer vision (Vol. 4, pp. 298–311). Google Scholar
  46. Landis, H. (2002). Production-ready global illumination. In L. Gritz (Ed.), RenderMan in production: SIGGRAPH 2002 course 16 (pp. 87–100). New York: ACM. Google Scholar
  47. Le Besnerais, G., & Champagnat, F. (2005). Dense optical flow by iterative local window registration. In Proceedings of the international conference on image processing (Vol. 1, pp. 137–140). Google Scholar
  48. Lei, C., & Yang, Y. (2009). Optical flow estimation on coarse-to-fine region-trees using discrete optimization. In Proceedings of the IEEE international conference on computer vision. Google Scholar
  49. Lempitsky, V., Roth, S., & Rother, C. (2008). Fusion flow: discrete-continuous optimization for optical flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  50. Levoy, M. (1988). Display of surfaces from volume data. IEEE Computer Graphics and Applications, 8(3), 29–37. CrossRefGoogle Scholar
  51. Li, Y., & Huttenlocher, D. (2008). Learning for optical flow using stochastic optimization. In Proceedings of the European conference on computer vision (Vol. 2, pp. 373–391). Google Scholar
  52. Liu, C., Freeman, W., Adelson, E., & Weiss, Y. (2008). Human-assisted motion annotation. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  53. Liu, C., Yuen, J., Torralba, A., Sivic, J., & Freeman, W. (2008). SIFT flow: dense correspondence across difference scenes. In Proceedings of the European conference on computer vision (Vol. 3, pp. 28–42). Google Scholar
  54. Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application in stereo vision. In Proceedings of the international joint conference on artificial intelligence (pp. 674–679). Google Scholar
  55. Mahajan, D., Huang, F., Matusik, W., Ramamoorthi, R., & Belhumeur, P. (2009). Moving gradients: a path-based method for plausible image interpolation. In Annual conference series. ACM computer graphics, SIGGRAPH. Google Scholar
  56. Markandey, V., & Flinchbaugh, B. (1990). Multispectral constraints for optical flow computation. In Proceedings of the IEEE international conference on computer vision (pp. 38–41). Google Scholar
  57. McCane, B., Novins, K., Crannitch, D., & Galvin, B. (2001). On benchmarking optical flow. Computer Vision and Image Understanding, 84(1), 126–143. MATHCrossRefGoogle Scholar
  58. Mitiche, A., & Bouthemy, P. (1996). Computation and analysis of image motion: a synopsis of current problems and methods. International Journal of Computer Vision, 19(1), 29–55. CrossRefGoogle Scholar
  59. Mova LLC (2004). Contour reality capture. http://www.mova.com/.
  60. Murray, D., & Buxton, B. (1987). Scene segmentation from visual motion using global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2), 220–228. CrossRefGoogle Scholar
  61. Nagel, H.-H., & Enkelmann, W. (1986). An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(5), 565–593. CrossRefGoogle Scholar
  62. Negahdaripour, S. (1998). Revised definition of optical flow: integration of radiometric and geometric cues for dynamic scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(9), 961–979. CrossRefGoogle Scholar
  63. Nir, T., Bruckstein, A., & Kimmel, R. (2008). Over-parameterized variational optical flow. International Journal of Computer Vision, 76(2), 205–216. CrossRefGoogle Scholar
  64. Ohta, N. (1989). Optical flow detection by color images. In International conference on image processing (pp. 801–805). Google Scholar
  65. Otte, M., & Nagel, H.-H. (1994). Optical flow estimation: advances and comparisons. In Proceedings of the European conference on computer vision (pp. 51–60). Google Scholar
  66. Philips, P., Scruggs, W., O’Toole, A., Flynn, P., Bowyer, K., Schott, C., & Sharpe, M. (2005). Overview of the face recognition grand challenge. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 947–954). Google Scholar
  67. Pock, T., Pock, M., & Bischof, H. (2007). Algorithmic differentiation: application to variational problems in computer vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(7), 1180–1193. CrossRefGoogle Scholar
  68. Pratt, W. (1974). Correlation techniques of image registration. IEEE Transactions on Aerospace and Electronic Systems, AES-10, 353–358. CrossRefGoogle Scholar
  69. Ramnath, K., Baker, S., Matthews, I., & Ramanan, D. (2008). Increasing the density of active appearance models. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  70. Rannacher, J. (2009). Realtime 3D motion estimation on graphics hardware. Undergraduate thesis, Heidelberg University. Google Scholar
  71. Ren, X. (2008). Local grouping for optical flow. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  72. Roth, S., & Black, M. (2007). On the spatial statistics of optical flow. International Journal of Computer Vision, 74(1), 33–50. CrossRefGoogle Scholar
  73. Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(13), 7–42. MATHCrossRefGoogle Scholar
  74. Scharstein, D., & Szeliski, R. (2003). High-accuracy stereo depth maps using structured light. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 195–202). Google Scholar
  75. Seitz, S., & Baker, S. (2009). Filter flow. In Proceedings of the IEEE international conference on computer vision. Google Scholar
  76. Seitz, S., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 519–526). Google Scholar
  77. Shade, J., Gortler, S., He, L.-W., & Szeliski, R. (1998). Layered depth images. In Annual conference series. ACM computer graphics, SIGGRAPH (pp. 231–242). Google Scholar
  78. Shizawa, M., & Mase, K. (1991). A unified computational theory for motion transparency and motion boundaries based on eigenenergy analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 289–295). CrossRefGoogle Scholar
  79. Sim, T., Baker, S., & Bsat, M. (2003). The CMU pose, illumination, and expression database. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12), 1615–1618. CrossRefGoogle Scholar
  80. Stiller, C., & Konrad, J. (1999). Estimating motion in image sequences: a tutorial on modeling and computation of 2D motion. IEEE Signal Processing Magazine, 16(4), 70–91. CrossRefGoogle Scholar
  81. Sun, C. (1999). Fast optical flow using cross correlation and shortest-path techniques. In Proceedings of digital image computing: techniques and applications (pp. 143–148). Google Scholar
  82. Sun, J., Shum, H.-Y., & Zheng, N. (2003). Stereo matching using belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(7), 787–800. CrossRefGoogle Scholar
  83. Sun, D., Roth, S., Lewis, J., & Black, M. (2008). Learning optical flow. In Proceedings of the European conference on computer vision (Vol. 3, pp. 83–97). Google Scholar
  84. Sun, D., Roth, S., & Black, M. (2010). Secrets of optical flow estimation and their principles. In Proceedings of the IEEE conference on computer vision and pattern recognition. Google Scholar
  85. Szeliski, R. (1999). Prediction error as a quality metric for motion and stereo. In Proceedings of the IEEE international conference on computer vision (pp. 781–788). CrossRefGoogle Scholar
  86. Tappen, M., Adelson, E., & Freeman, W. (2006). Estimating intrinsic component images using non-linear regression. In Proceedings of the IEEE conference on computer vision and pattern recognition (Vol. 2, pp. 1992–1999). Google Scholar
  87. Trobin, W., Pock, T., Cremers, D., & Bischof, H. (2008). Continuous energy minimization via repeated binary fusion. In Proceedings of the European conference on computer vision (Vol. 4, pp. 677–690). Google Scholar
  88. Trobin, W., Pock, T., Cremers, D., & Bischof, H. (2008). An unbiased second-order prior for high-accuracy motion estimation. In Proceedings of pattern recognition, DAGM (pp. 396–405). Google Scholar
  89. Valgaerts, L., Bruhn, A., & Weickert, J. (2008). A variational model for the joint recovery of the fundamental matrix and the optical flow. In Proceedings of pattern recognition, DAGM (pp. 314–324). Google Scholar
  90. Vedula, S., Baker, S., Rander, P., Collins, R., & Kanade, T. (2005). Three-dimensional scene flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3), 475–480. CrossRefGoogle Scholar
  91. Wang, J., & Adelson, E. (1993). Layered representation for motion analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 361–366). CrossRefGoogle Scholar
  92. Wedel, A., Pock, T., Braun, J., Franke, U., & Cremers, D. (2008). Duality TV-L1 flow with fundamental matrix prior. In Proceedings of image and vision computing, New Zealand. Google Scholar
  93. Wedel, A., Pock, T., Zach, C., Cremers, D., & Bischof, H. (2008). An improved algorithm for TV-L1 optical flow. In Proceedings of the Dagstuhl motion workshop. Google Scholar
  94. Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In Proceedings of the IEEE international conference on computer vision. Google Scholar
  95. Weiss, Y. (1997). Smoothness in layers: motion segmentation using nonparametric mixture estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 520–526). CrossRefGoogle Scholar
  96. Werlberger, M., Trobin, W., Pock, T., Bischof, H., Wedel, A., & Cremers, D. (2009). Anisotropic Huber-L1 optical flow. In Proceedings of the British machine vision conference. Google Scholar
  97. Xu, L., Chen, J., & Jia, J. (2008). A segmentation based variational model for accurate optical flow estimation. In Proceedings of the European conference on computer vision (Vol. 1, pp. 671–684). Google Scholar
  98. Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H.-P. (2009). Complementary optic flow. In Proceedings of seventh international workshop on energy minimization methods in computer vision and pattern recognition. Google Scholar
  99. Zitnick, C., Kang, S., Uyttendaele, M., Winder, S., & Szeliski, R. (2004). High-quality video view interpolation using a layered representation. In Annual conference series: Vol. 23(2). ACM computer graphics, SIGGRAPH (pp. 600–608). CrossRefGoogle Scholar

Copyright information

© The Author(s) 2010

Authors and Affiliations

  • Simon Baker
    • 5
  • Daniel Scharstein
    • 1
  • J. P. Lewis
    • 2
  • Stefan Roth
    • 3
  • Michael J. Black
    • 4
  • Richard Szeliski
    • 5
  1. 1.Middlebury CollegeMiddleburyUSA
  2. 2.Weta DigitalWellingtonNew Zealand
  3. 3.TU DarmstadtDarmstadtGermany
  4. 4.Brown UniversityProvidenceUSA
  5. 5.Microsoft ResearchRedmondUSA

Personalised recommendations