A Naturalistic Open Source Movie for Optical Flow Evaluation

  • Daniel J. Butler
  • Jonas Wulff
  • Garrett B. Stanley
  • Michael J. Black
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7577)


Ground truth optical flow is difficult to measure in real scenes with natural motion. As a result, optical flow data sets are restricted in terms of size, complexity, and diversity, making optical flow algorithms difficult to train and test on realistic data. We introduce a new optical flow data set derived from the open source 3D animated short film Sintel. This data set has important features not present in the popular Middlebury flow evaluation: long sequences, large motions, specular reflections, motion blur, defocus blur, and atmospheric effects. Because the graphics data that generated the movie is open source, we are able to render scenes under conditions of varying complexity to evaluate where existing flow algorithms fail. We evaluate several recent optical flow algorithms and find that current highly-ranked methods on the Middlebury evaluation have difficulty with this more complex data set suggesting further research on optical flow estimation is needed. To validate the use of synthetic data, we compare the image- and flow-statistics of Sintel to those of real films and videos and show that they are similar. The data set, metrics, and evaluation website are publicly available.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. IJCV 92, 1–31 (2011)CrossRefGoogle Scholar
  2. 2.
    Roosendaal, T. (Producer): Sintel. Blender Foundation, Durian Open Movie Project (2010),
  3. 3.
  4. 4.
    Butler, D., Wulff, J., Stanley, G., Black, M.: MPI-Sintel optical flow benchmark: Supplemental material. MPI-IS-TR-006, MPI for Intelligent Systems (2012)Google Scholar
  5. 5.
    Roth, S., Black, M.: On the spatial statistics of optical flow. IJCV 74, 33–50 (2007)CrossRefGoogle Scholar
  6. 6.
    Field, D.: Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A 4, 2379–2394 (1987)CrossRefGoogle Scholar
  7. 7.
    Simoncelli, E., Olshausen, B.: Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 1193–1216 (2001)CrossRefGoogle Scholar
  8. 8.
  9. 9.
    Barron, J., Fleet, D., Beauchemin, S.: Performance of optical flow techniques. IJCV 12, 43–77 (1994)CrossRefGoogle Scholar
  10. 10.
    McCane, B., Novins, K., Crannitch, D., Galvin, B.: On benchmarking optical flow. CVIU 84, 126–143 (2001)zbMATHGoogle Scholar
  11. 11.
    Otte, M., Nagel, H.-H.: Optical Flow Estimation: Advances and Comparisons. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 800, pp. 51–60. Springer, Heidelberg (1994)Google Scholar
  12. 12.
    Liu, C., Freeman, W., Adelson, E., Weiss, Y.: Human-assisted motion annotation. In: CVPR, pp. 1–8 (2008)Google Scholar
  13. 13.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: CVPR, pp. 3354–3361 (2012)Google Scholar
  14. 14.
    Meister, S., Jaehne, B., Kondermann, D.: An outdoor stereo camera system for the generation of real-world benchmark datasets. Opt. Eng. 51, 021107 (2012)Google Scholar
  15. 15.
    Sun, D., Roth, S., Lewis, J.P., Black, M.J.: Learning Optical Flow. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 83–97. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Brox, T., Bregler, C., Malik, J.: Large displacement optical flow: Descriptor matching in variational motion estimation. PAMI 33, 500–513 (2009)CrossRefGoogle Scholar
  17. 17.
    Sun, D., Roth, S., Black, M.: Secrets of optical flow estimation and their principles. In: CVPR, pp. 2432–2439 (2010)Google Scholar
  18. 18.
    Horn, B., Schunck, B.: Determining optical flow. AIJ 16, 185–203 (1981)Google Scholar
  19. 19.
    Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., Bischof, H.: Anisotropic Huber-L1 optical flow. In: BMVC, pp. 1–11 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Daniel J. Butler
    • 1
  • Jonas Wulff
    • 2
  • Garrett B. Stanley
    • 3
  • Michael J. Black
    • 2
  1. 1.University of WashingtonSeattleUSA
  2. 2.Max-Planck Institute for Intelligent SystemsTübingenGermany
  3. 3.Georgia Institute of TechnologyAtlantaUSA

Personalised recommendations