Scene Flow Estimation from Light Fields via the Preconditioned Primal-Dual Algorithm

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8753)


In this paper we present a novel variational model to jointly estimate geometry and motion from a sequence of light fields captured with a plenoptic camera. The proposed model uses the so-called sub-aperture representation of the light field. Sub-aperture images represent images with slightly different viewpoints, which can be extracted from the light field. The sub-aperture representation allows us to formulate a convex global energy functional, which enforces multi-view geometry consistency, and piecewise smoothness assumptions on the scene flow variables. We optimize the proposed scene flow model by using an efficient preconditioned primal-dual algorithm. Finally, we also present synthetic and real world experiments.


Scene Flow Light Field Capture Plenoptic Camera Data Fidelity Term Lytro Camera 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Adelson, E.H., Wang, J.Y.A.: Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 99–106 (1992)CrossRefGoogle Scholar
  2. 2.
    Alvertos, P., Patras, I., Alvertos, N., Tziritas, G.: Joint disparity and motion field estimation in stereoscopic image sequences. In: 13th International Conference on Pattern Recognition, pp. 359–362 (1996)Google Scholar
  3. 3.
    Basha, T., Moses, Y., Kiryati, N.: Multi-view scene flow estimation: a view centered variational approach. Int. J. Comput. Vis. 101(1), 6–21 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bishop, T., Favaro, P.: Plenoptic depth estimation from multiple aliased views. In: 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1622–1629. IEEE (2009)Google Scholar
  5. 5.
    Bishop, T.E., Favaro, P.: The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 972–986 (2012)CrossRefGoogle Scholar
  6. 6.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    Carceroni, R.L., Kutulakos, K.N.: Multi-view scene capture by surfel sampling: from video streams to non-rigid 3d motion, shape and reflectance. Int. J. Comput. Vis. 49, 175–214 (2002)CrossRefzbMATHGoogle Scholar
  9. 9.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Courchay, J., Pons, J.-P., Monasse, P., Keriven, R.: Dense and accurate spatio-temporal multi-view stereovision. In: Zha, H., Taniguchi, R., Maybank, S. (eds.) ACCV 2009, Part II. LNCS, vol. 5995, pp. 11–22. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Furukawa, Y., Ponce, J.: Dense 3d motion capture from synchronized video streams. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2008). IEEE Computer Society, Anchorage, 24–26 June 2008Google Scholar
  12. 12.
    Goldluecke, B., Wanner, S.: The variational structure of disparity and regularization of 4d light fields. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  13. 13.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: SIGGRAPH, pp. 43–54 (1996)Google Scholar
  14. 14.
    Heber, S., Ranftl, R., Pock, T.: Variational shape from light field. In: International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (2013)Google Scholar
  15. 15.
    Isaksen, A., McMillan, L., Gortler, S.J.: Dynamically reparameterized light fields. In: SIGGRAPH, pp. 297–306 (2000)Google Scholar
  16. 16.
    Keriven, R., Faugeras, O.: Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. Int. J. Comput. Vis. 72, 2007 (2006)Google Scholar
  17. 17.
    Lumsdaine, A., Georgiev, T.: The focused plenoptic camera. In. Proceedings of IEEE ICCP, pp. 1–8 (2009)Google Scholar
  18. 18.
    Neumann, J., Aloimonos, Y.: Spatio-temporal stereo using multi-resolution subdivision surfaces. Int. J. Comput. Vis. 47, 2002 (2002)CrossRefGoogle Scholar
  19. 19.
    Ng, R.: Digital light field photography. Ph.D. thesis, Stanford University (2006).
  20. 20.
    Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: International Conference on Computer Vision (ICCV), pp. 1762–1769. IEEE (2011)Google Scholar
  21. 21.
    Ranftl, R., Gehrig, S., Pock, T., Bischof, H.: Pushing the limits of stereo using variational stereo estimation. In: Intelligent Vehicles Symposium, pp. 401–407. IEEE (2012)Google Scholar
  22. 22.
    Vedula, S., Baker, S., Rander, P., Collins, R.T., Kanade, T.: Three-dimensional scene flow. In: ICCV, pp. 722–729 (1999)Google Scholar
  23. 23.
    Wanner, S., Goldluecke, B.: Globally consistent depth labeling of 4D lightfields. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  24. 24.
    Wanner, S., Goldluecke, B.: Spatial and angular variational super-resolution of 4D light fields. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 608–621. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  25. 25.
    Wanner, S., Straehle, C., Goldluecke, B.: Globally consistent multi-label assignment on the ray space of 4d light fields. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  26. 26.
    Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An Improved Algorithm for TV-\({L}^{1}\) Optical Flow. In: Cremers, D., Rosenhahn, B., Yuille, A.L., Schmidt, F.R. (eds.) Statistical and Geometrical Approaches to Visual Motion Analysis. LNCS, vol. 5604, pp. 23–45. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  27. 27.
    Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., Cremers, D.: Efficient dense 3d scene flow from sparse or dense stereo data, Oct 2008Google Scholar
  28. 28.
    Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Trans. Graph. 24(3), 765–776 (2005)CrossRefGoogle Scholar
  29. 29.
    Zhang, Y., Kambhamettu, C., Kambhamettu, R.: On 3d scene flow and structure estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 778–785 (2001)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  1. 1.Institute for Computer Graphics and VisionGraz University of TechnologyGrazAustria
  2. 2.Safety and Security DepartmentAIT Austrian Institute of TechnologySeibersdorfAustria

Personalised recommendations