Depth Estimation Through a Generative Model of Light Field Synthesis

  • Mehdi S. M. Sajjadi
  • Rolf Köhler
  • Bernhard Schölkopf
  • Michael Hirsch
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9796)

Abstract

Light field photography captures rich structural information that may facilitate a number of traditional image processing and computer vision tasks. A crucial ingredient in such endeavors is accurate depth recovery. We present a novel framework that allows the recovery of a high quality continuous depth map from light field data. To this end we propose a generative model of a light field that is fully parametrized by its corresponding depth map. The model allows for the integration of powerful regularization techniques such as a non-local means prior, facilitating accurate depth map estimation. Comparisons with previous methods show that we are able to recover faithful depth maps with much finer details. In a number of challenging real-world examples we demonstrate both the effectiveness and robustness of our approach.

References

  1. 1.
    The (new) stanford light field archive (2008). http://lightfield.stanford.edu. Accessed 07 Apr 2016
  2. 2.
    Adelson, E.H., Wang, J.Y.A.: Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 14(2), 99–106 (1992)CrossRefGoogle Scholar
  3. 3.
    Bishop, T.E., Favaro, P.: The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 34(5), 972–986 (2012)CrossRefGoogle Scholar
  4. 4.
    Bolles, R.C., Baker, H.H., Marimont, D.H.: Epipolar-plane image analysis: an approach to determining structure from motion. Int. J. Comput. Vis. 1(1), 7–55 (1987)CrossRefGoogle Scholar
  5. 5.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: CVPR (2005)Google Scholar
  6. 6.
    Carbonetto, P.: A programming interface for L-BFGS-B in MATLAB (2014). https://github.com/pcarbo/lbfgsb-matlab. Accessed 15 Apr 2015
  7. 7.
    Chai, J.X., Tong, X., Chan, S.C., Shum, H.Y.: Plenoptic sampling. In: ACM SIGGRAPH (2000)Google Scholar
  8. 8.
    Cho, D., Kim, S., Tai, Y.-W.: Consistent matting for light field images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part IV. LNCS, vol. 8692, pp. 90–104. Springer, Heidelberg (2014)Google Scholar
  9. 9.
    Dansereau, D.G., Bongiorno, D.L., Pizarro, O., Williams, S.B.: Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter. In: IS&T/SPIE Electronic Imaging (2013)Google Scholar
  10. 10.
    Dansereau, D.G., Mahon, I., Pizarro, O., Williams, S.B.: Plenoptic flow: closed-form visual odometry for light field cameras. In: IROS (2011)Google Scholar
  11. 11.
    Dansereau, D.G., Pizarro, O., Williams, S.B.: Decoding, calibration and rectification for lenselet-based plenoptic cameras. In: CVPR (2013)Google Scholar
  12. 12.
    Diebold, M., Goldlücke, B.: Epipolar plane image refocusing for improved depth estimation and occlusion handling. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2013)Google Scholar
  13. 13.
    Favaro, P.: Recovering thin structures via nonlocal-means regularization with application to depth from defocus. In: CVPR (2010)Google Scholar
  14. 14.
    Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: ICCV (2013)Google Scholar
  15. 15.
    Goldluecke, B., Wanner, S.: The variational structure of disparity and regularization of 4D light fields. In: CVPR (2013)Google Scholar
  16. 16.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: ACM SIGGRAPH (1996)Google Scholar
  17. 17.
    Heber, S., Pock, T.: Shape from light field meets robust PCA. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 751–767. Springer, Heidelberg (2014)Google Scholar
  18. 18.
    Heber, S., Ranftl, R., Pock, T.: Variational shape from light field. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.-C. (eds.) EMMCVPR 2013. LNCS, vol. 8081, pp. 66–79. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  19. 19.
    Isaksen, A., McMillan, L., Gortler, S.J.: Dynamically reparameterized light fields. In: ACM SIGGRAPH. ACM (1996)Google Scholar
  20. 20.
    Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.H.: Scene reconstruction from high spatio-angular resolution light fields. ACM SIGGRAPH (2013)Google Scholar
  21. 21.
    Levoy, M., Hanrahan, P.: Light field rendering. In: ACM SIGGRAPH. ACM (1996)Google Scholar
  22. 22.
    Li, N., Ye, J., Ji, Y., Ling, H., Yu, J.: Saliency detection on light field. In: CVPR (2014)Google Scholar
  23. 23.
    Liang, C.K., Lin, T.H., Wong, B.Y., Liu, C., Chen, H.H.: Programmable aperture photography: multiplexed light field acquisition. ACM SIGGRAPH (2008)Google Scholar
  24. 24.
    Lin, H., Chen, C., Bing Kang, S., Yu, J.: Depth recovery from light field using focal stack symmetry. In: ICCV (2015)Google Scholar
  25. 25.
    Ng, R.: Digital light field photography. Ph.D. thesis, stanford university (2006). Ren Ng founded LytroGoogle Scholar
  26. 26.
    Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR 2(11) (2005)Google Scholar
  27. 27.
    Park, J., Kim, H., Tai, Y.W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3D-TOF cameras. In: ICCV (2011)Google Scholar
  28. 28.
    Perwass, C., Wietzke, L.: The next generation of photography (2010). https://github.com/pcarbo/lbfgsb-matlab. Accessed 15 Apr 2015, Perwass and Wietzke founded Raytrix
  29. 29.
    Perwass, C., Wietzke, L.: Single lens 3D-camera with extended depth-of-field. In: IS&T/SPIE Electronic Imaging (2012)Google Scholar
  30. 30.
    Sebe, I.O., Ramanathan, P., Girod, B.: Multi-view geometry estimation for light field compression. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2002)Google Scholar
  31. 31.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: CVPR (2010)Google Scholar
  32. 32.
    Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: ICCV (2013)Google Scholar
  33. 33.
    Tosic, I., Berkner, K.: Light field scale-depth space transform for dense depth estimation. In: CVPR Workshops (2014)Google Scholar
  34. 34.
    Vaish, V., Wilburn, B., Joshi, N., Levoy, M.: Using plane + parallax for calibrating dense camera arrays. In: CVPR (2004)Google Scholar
  35. 35.
    Wang, T.C., Efros, A.A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: ICCV (2015)Google Scholar
  36. 36.
    Wanner, S., Goldluecke, B.: Globally consistent depth labeling of 4D light fields. In: CVPR (2012)Google Scholar
  37. 37.
    Wanner, S., Goldluecke, B.: Spatial and angular variational super-resolution of 4D light fields. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 608–621. Springer, Heidelberg (2012)Google Scholar
  38. 38.
    Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 36(3), 606–619 (2014)CrossRefGoogle Scholar
  39. 39.
    Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2013)Google Scholar
  40. 40.
    Wanner, S., Straehle, C., Goldluecke, B.: Globally consistent multi-label assignment on the ray space of 4D light fields. In: CVPR (2013)Google Scholar
  41. 41.
    Zhang, Z., Liu, Y., Dai, Q.: Light field from micro-baseline image pair. In: CVPR (2015)Google Scholar
  42. 42.
    Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B: fortran subroutines for large-scale bound-constrained optimization. ACM TOMS 23(4), 550–560 (1997)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Mehdi S. M. Sajjadi
    • 1
  • Rolf Köhler
    • 1
  • Bernhard Schölkopf
    • 1
  • Michael Hirsch
    • 1
  1. 1.Max-Planck-Institute for Intelligent SystemsTübingenGermany

Personalised recommendations