Advertisement

Differential Scene Flow from Light Field Gradients

  • Sizhuo Ma
  • Brandon M. Smith
  • Mohit GuptaEmail author
Article
  • 73 Downloads

Abstract

This paper presents novel techniques for recovering 3D dense scene flow, based on differential analysis of 4D light fields. The key enabling result is a per-ray linear equation, called the ray flow equation, that relates 3D scene flow to 4D light field gradients. The ray flow equation is invariant to 3D scene structure and applicable to a general class of scenes, but is under-constrained (3 unknowns per equation). Thus, additional constraints must be imposed to recover motion. We develop two families of scene flow algorithms by leveraging the structural similarity between ray flow and optical flow equations: local ‘Lucas–Kanade’ ray flow and global ‘Horn–Schunck’ ray flow, inspired by corresponding optical flow methods. We also develop a combined local–global method by utilizing the correspondence structure in the light fields. We demonstrate high precision 3D scene flow recovery for a wide range of scenarios, including rotation and non-rigid motion. We analyze the theoretical and practical performance limits of the proposed techniques via the light field structure tensor, a \(3 \times 3\) matrix that encodes the local structure of light fields. We envision that the proposed analysis and algorithms will lead to design of future light-field cameras that are optimized for motion sensing, in addition to depth sensing.

Keywords

Scene flow 3D motion estimation Differential analysis Differential motion Light fields 3D shape and motion estimation Computational cameras 

Notes

Supplementary material

Supplementary material 1 (mp4 23017 KB)

11263_2019_1230_MOESM2_ESM.pdf (1.4 mb)
Supplementary material 2 (pdf 1402 KB)

References

  1. Adelson, E. H., & Wang, J. Y. A. (1992). Single lens stereo with a plenoptic camera. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 14(2), 99–106.CrossRefGoogle Scholar
  2. Alexander, E., Guo, Q., Koppal, S., Gortler, S., & Zickler, T. (2016). Focal flow: Measuring distance and velocity with defocus and differential motion. In European conference on computer vision (ECCV) (pp. 667–682). Heidelberg: Springer.Google Scholar
  3. Aujol, J. F., Gilboa, G., Chan, T., & Osher, S. (2006). Structure-texture image decomposition-modeling, algorithms, and parameter selection. International Journal of Computer Vision (IJCV), 67(1), 111–136.  https://doi.org/10.1007/s11263-006-4331-z.CrossRefzbMATHGoogle Scholar
  4. Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104.CrossRefGoogle Scholar
  5. Bok, Y., Jeon, H. G., & Kweon, I. S. (2017). Geometric calibration of micro-lens-based light field cameras using line features. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 39(2), 287–300.CrossRefGoogle Scholar
  6. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision (ECCV) (pp. 25–36).Google Scholar
  7. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2014). High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision (ECCV) (Vol. 3024, pp. 25–36). Springer.Google Scholar
  8. Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision (IJCV), 61(3), 211–231.CrossRefGoogle Scholar
  9. Chandraker, M. (2014a). On shape and material recovery from motion. In European conference on computer vision (ECCV) (pp. 202–217). Heidelberg: Springer.Google Scholar
  10. Chandraker, M. (2014b). What camera motion reveals about shape with unknown BRDF. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2171–2178). Washington: IEEE.Google Scholar
  11. Chandraker, M. (2016). The information available to a moving observer on shape with unknown, isotropic brdfs. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 38(7), 1283–1297.CrossRefGoogle Scholar
  12. Dansereau, D. G., Mahon, I., Pizarro, O., & Williams, S. B. (2011). Plenoptic flow: Closed-form visual odometry for light field cameras. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4455–4462). Washington: IEEE.Google Scholar
  13. Dansereau, D. G., Schuster, G., Ford, J., & Wetzstein, G. (2017). A wide-field-of-view monocentric light field camera. In IEEE conference on computer vision and pattern recognition (CVPR). Washington: IEEE.Google Scholar
  14. Gottfried, J. M., Fehr, J., & Garbe, C. S. (2011). Computing range flow from multi-modal kinect data. In International symposium on visual computing (pp. 758–767). Heidelberg: Springer.Google Scholar
  15. Hasinoff, S. W., Durand, F., & Freeman, W. T. (2010). Noise-optimal capture for high dynamic range photography. In: IEEE conference on computer vision and pattern recognition (CVPR) (pp. 553–560). IEEE.Google Scholar
  16. Haussecker, H. W., & Fleet, D. J. (2001). Computing optical flow with physical models of brightness variation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 661–673.CrossRefGoogle Scholar
  17. Heber, S., & Pock, T. (2014). Scene flow estimation from light fields via the preconditioned primal–dual algorithm. In X. Jiang, J. Hornegger, & R. Koch (Eds.), Pattern recognition (pp. 3–14). Cham: Springer International.Google Scholar
  18. Horn, B. K., & Schunck, B. G. (1981). Determining optical flow. Artificial Intelligence, 17(1–3), 185–203.CrossRefGoogle Scholar
  19. Hung, C. H., Xu, L., & Jia, J. (2013). Consistent binocular depth and scene flow with chained temporal profiles. International Journal of Computer Vision (IJCV), 102(1–3), 271–292.CrossRefzbMATHGoogle Scholar
  20. Jaimez, M., Souiai, M., Gonzalez-Jimenez, J., & Cremers, D. (2015). A primal-dual framework for real-time dense rgb-d scene flow. In IEEE international conference on robotics and automation (ICRA) (pp. 98–104). Washington: IEEE.Google Scholar
  21. Jo, K., Gupta, M., & Nayar, S. K. (2015). SpeDo: 6 DOF ego-motion sensor using speckle defocus imaging. InIEEE international conference on computer vision (ICCV) (pp. 4319–4327). Washington: IEEE.Google Scholar
  22. Johannsen, O., Sulc, A., & Goldluecke, B. (2015). On linear structure from motion for light field cameras. In IEEE international conference on computer vision (ICCV) (pp. 720–728). Washington: IEEE.Google Scholar
  23. Letouzey, A., Petit, B., & Boyer, E. (2011). Scene flow from depth and color images. In British machine vision conference (BMVC) (pp. 46–56). BMVA Press.Google Scholar
  24. Levoy, M., & Hanrahan, P. (1996). Light field rendering. In SIGGRAPH conference on computer graphics and interactive techniques (pp. 31–42). New York: ACM.Google Scholar
  25. Li, Z., Xu, Z., Ramamoorthi, R., & Chandraker, M. (2017). Robust energy minimization for BRDF-invariant shape from light fields. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 1). Washington: IEEE.Google Scholar
  26. Lucas, B. D., Kanade, T., et al. (1981). An iterative image registration technique with an application to stereo vision. In International joint conference on artificial intelligence (pp. 674–679). San Francisco: Morgan Kaufmann.Google Scholar
  27. Ma, S., Smith, B. M., & Gupta, M. (2018). 3D scene flow from 4D light field gradients. In European conference on computer vision (ECCV) (Vol. 8, pp. 681–698). Springer International Publishing.Google Scholar
  28. Navarro, J., & Garamendi, J. (2016). Variational scene flow and occlusion detection from a light field sequence. In International conference on systems, signals and image processing (IWSSIP) (pp. 1–4). Washington: IEEE.Google Scholar
  29. Neumann, J., Fermuller, C., & Aloimonos, Y. (2003). Polydioptric camera design and 3d motion estimation. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. 2, pp. II–294). Washington: IEEE.Google Scholar
  30. Neumann, J., Fermüller, C., & Aloimonos, Y. (2004). A hierarchy of cameras for 3d photography. Computer Vision and Image Understanding, 96(3), 274–293.CrossRefGoogle Scholar
  31. Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., & Hanrahan, P. (2005). Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR, 2(11), 1–11.Google Scholar
  32. Odobez, J. M., & Bouthemy, P. (1995). Robust multiresolution estimation of parametric motion models. Journal of Visual Communication and Image Representation, 6(4), 348–365.CrossRefGoogle Scholar
  33. Phong, B. T. (1975). Illumination for computer generated pictures. Communications of the ACM, 18(6), 311–317.  https://doi.org/10.1145/360825.360839.CrossRefGoogle Scholar
  34. Ranjan, A., Jampani, V., Balles, L., Kim, K., Sun, D., Wulff, J., et al. (2018). Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. arXiv: 1805.09806 [cs]
  35. Shi, J., & Tomasi, C. (1994). Good features to track. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 593–600). Washington: IEEE.Google Scholar
  36. Smith, B., O’Toole, M., & Gupta, M. (2018). Tracking multiple objects outside the line of sight using speckle imaging. In IEEE conference on computer vision and pattern recognition (CVPR). IEEE.Google Scholar
  37. Smith, B. M., Desai, P., Agarwal, V., & Gupta, M. (2017). CoLux: Multi-object 3d micro-motion analysis using speckle imaging. ACM Transactions on Graphics, 36(4), 1–12.Google Scholar
  38. Srinivasan, P. P., Tao, M. W., Ng, R., & Ramamoorthi, R. (2015). Oriented light-field windows for scene flow. In IEEE international conference on computer vision (ICCV) (pp. 3496–3504). Washington: IEEE.Google Scholar
  39. Sun, D., Roth, S., & Black, M. J. (2010). Secrets of optical flow estimation and their principles. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2432–2439). Washington, IEEE.Google Scholar
  40. Sun, D., Sudderth, E. B., & Pfister, H. (2015). Layered RGBD scene flow estimation. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 548–556). Washington: IEEE.Google Scholar
  41. Tao, M. W., Hadap, S., Malik, J., & Ramamoorthi, R. (2013). Depth from combining defocus and correspondence using light-field cameras. In IEEE international conference on computer vision (ICCV) (pp. 673–680). Washington: IEEE.Google Scholar
  42. Vedula, S., Baker, S., Rander, P., Collins, R., & Kanade, T. (1999). Three-dimensional scene flow. In IEEE international conference on computer vision (ICCV) (Vol. 2, pp. 722–729). Washington: IEEE.Google Scholar
  43. Vijayanarasimhan, S., Ricco, S., Schmid, C., Sukthankar, R., & Fragkiadaki, K. (2017). SfM-Net: Learning of structure and motion from video. arXiv:1704.07804 [cs]
  44. Wang, T. C., Chandraker, M., Efros, A. A., & Ramamoorthi, R. (2016). SVBRDF-invariant shape and reflectance estimation from light-field cameras. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 5451–5459). Washington: IEEE.Google Scholar
  45. Wanner, S., & Goldluecke, B. (2014). Variational light field analysis for disparity estimation and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 36(3), 606–619.CrossRefGoogle Scholar
  46. Wedel, A., Rabe, C., Vaudrey, T., Brox, T., Franke, U., & Cremers, D. (2008). Efficient dense scene flow from sparse or dense stereo data. In European conference on computer vision (ECCV) (pp. 739–751). Heidelberg: Springer.Google Scholar
  47. Yin, Z., & Shi, J. (2018). GeoNet: Unsupervised learning of dense depth, optical flow and camera pose. In IEEE/CVF conference on computer vision and pattern recognition (pp. 1983–1992). Salt Lake City, UT: IEEE.  https://doi.org/10.1109/CVPR.2018.00212, https://ieeexplore.ieee.org/document/8578310/
  48. Zhang, Y., Li, Z., Yang, W., Yu, P., Lin, H., & Yu, J. (2017). The light field 3d scanner. In IEEE international conference on computational photography (ICCP) (pp. 1–9). Washington: IEEE.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer SciencesUniversity of Wisconsin-MadisonMadisonUSA

Personalised recommendations