Skip to main content
SpringerLink
Log in

3D Scene Reconstruction with an Un-calibrated Light Field Camera

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper is concerned with the problem of multi-view 3D reconstruction with an un-calibrated micro-lens array based light field camera. To acquire 3D Euclidean reconstruction, existing approaches commonly apply the calibration with a checkerboard and motion estimation from static scenes in two steps. Self-calibration is the process of simultaneously estimating intrinsic and extrinsic parameters directly from un-calibrated light fields without the help of a checkerboard. While the self-calibration technique for conventional (pinhole) camera is well understood, how to extend it to light field camera remains a challenging task. This is primarily due to the ultra-small baseline of the light field camera. We propose an effective self-calibration method for a light field camera for automatic metric reconstruction without a laborious pre-calibration process. In contrast to conventional self-calibration, we show how such a self-calibration method can be made numerically stable, by exploiting the regularity and measurement redundancies unique for the light field camera. The proposed method is built upon the derivation of a novel ray-space homography constraint (RSHC) using Plücker parameterization as well as a ray-space infinity homography (RSIH). We also propose a new concept of “rays of the absolute conic (RAC)” defined as a special quadric in 5D projective space \({\mathbb {P}}^5\). A set of new equations are established and solved for self-calibration and 3D metric reconstruction specifically designed for a light field camera . We validate the efficacy of the proposed method on both synthetic and real light fields, and have obtained superior results in both accuracy and robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Bartoli, A., & Sturm, P. (2001). The 3d line motion matrix and alignment of line reconstructions. In IEEE conference on computer vision and pattern recognition (CVPR) (Vol. I, pp. I–I). IEEE.

  • Bartoli, A., & Sturm, P. (2004). The 3d line motion matrix and alignment of line reconstructions. International Journal of Computer Vision, 57, 159–178.

    Article  Google Scholar 

  • Bartoli, A., & Sturm, P. (2005). Structure-from-motion using lines: Representation, triangulation, and bundle adjustment. Computer Vision and Image Understanding, 100(3), 416–441.

    Article  Google Scholar 

  • Birklbauer, C., & Bimber, O. (2014). Panorama light-field imaging. Computer Graphics Forum, 33(2), 43–52.

  • Bok, Y., Jeon, H. G., & Kweon, I. S. (2014). Geometric calibration of micro-lens-based light-field cameras using line features. In European conference on computer vision (ECCV) (pp 47–61).

  • Bok, Y., Jeon, H. G., & Kweon, I. S. (2017). Geometric calibration of micro-lens-based light field cameras using line features. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 39(2), 287–300

  • Chandraker, M., Agarwal, S., Kahl, F., Nistér, D., & Kriegman, D. (2007a). Autocalibration via rank-constrained estimation of the absolute quadric. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–8).

  • Chandraker, M., Agarwal, S., Kriegman, D., & Belongie, S. (2007b). Globally optimal affine and metric upgrades in stratified autocalibration. In IEEE international conference on computer vision (ICCV) (pp. 1–8).

  • Dansereau, D. G., Mahon, I., Pizarro, O., & Williams, S. B. (2011). Plenoptic flow: Closed-form visual odometry for light field cameras. In 2011 IEEE/RSJ international conference on intelligent robots and systems (pp. 4455–4462).

  • Dansereau, D. G., Pizarro, O., & Williams, S. B. (2013). Decoding, calibration and rectification for lenselet-based plenoptic cameras. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1027–1034).

  • Dong, F., Ieng, S. H., Savatier, X., Etienne-Cummings, R., & Benosman, R. (2013). Plenoptic cameras in real-time robotics. The International Journal of Robotics Research, 32(2), 206–217.

    Article  Google Scholar 

  • Faugeras, O. (1993). Three-dimensional computer vision: a geometric viewpoint. MIT Press.

  • Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.

    Article  MathSciNet  Google Scholar 

  • Gherardi, R., & Fusiello, A. (2010). Practical autocalibration. In European conference on computer vision (ECCV) (pp. 790–801).

  • Guo, X., Yu, Z., Kang, S. B., Lin, H., & Yu, J. (2016). Enhancing light fields through ray-space stitching. IEEE Transactions on Visualization and Computer Graphics, 22(7), 1852–1861.

    Article  Google Scholar 

  • Gurdjos, P., Bartoli, A., Sturm, P. (2009). Is dual linear self-calibration artificially ambiguous? In International conference on computer vision (ICCV) (pp. 88–95). IEEE.

  • Habed, A., Pani Paudel, D., Demonceaux, C., & Fofi, D. (2014). Efficient pruning LMI conditions for branch-and-prune rank and chirality-constrained estimation of the dual absolute quadric. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 493–500).

  • Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge University Press.

  • Hartley, R., Trumpf, J., Dai, Y., & Li, H. (2013). Rotation averaging. International Journal of Computer Vision, 103(3), 267–305.

    Article  MathSciNet  Google Scholar 

  • Hartley, R. I., & Sturm, P. (1997). Triangulation. Computer vision and image understanding, 68(2), 146–157.

    Article  Google Scholar 

  • Hartley, R. I., Hayman, E., de Agapito, L., & Reid, I. (1999). Camera calibration and the search for infinity. In IEEE international conference on computer vision (ICCV) (pp. 510–517).

  • Johannsen, O., Sulc, A., & Goldluecke, B. (2015). On linear structure from motion for light field cameras. In IEEE international conference on computer vision (ICCV) (pp. 720–728).

  • Johannsen, O., Sulc, A., Marniok, N., & Goldluecke, B. (2016). Layered scene reconstruction from multiple light field camera views. In Asian conference on computer vision (ACCV) (pp. 3–18)

  • Kneip, L., & Li, H. (2014). Efficient computation of relative pose for multi-camera systems. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 446–453).

  • Larsson, V., Kukelova, Z., & Zheng, Y. (2018). Camera pose estimation with unknown principal point. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2984–2992).

  • Levoy, M., & Hanrahan, P. (1996). Light field rendering. In Proceedings of the 23rd annual conference on computer graphics and interactive techniques (pp. 31–42).

  • Li, H., Hartley, R., & Kim, J. H. (2008). A linear approach to motion estimation using generalized camera models. In IEEE conference on computer vision and pattern recognition (CVPR) (pp 1–8).

  • Li, Y., Zhang, Q., Wang, X., & Wang, Q. (2019). Light field slam based on ray-space projection model. In Optoelectronic imaging and multimedia technology VI (Vol. 11187, p 1118706).

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.

    Article  Google Scholar 

  • Luong, Q. T., & Faugeras, O. D. (1997). Self-calibration of a moving camera from point correspondences and fundamental matrices. International Journal of Computer Vision, 22(3), 261–289.

    Article  Google Scholar 

  • Lytro. (2011). Lytro redefines photography with light field cameras. http://www.lytro.com.

  • Maybank, S. J., & Faugeras, O. D. (1992). A theory of self-calibration of a moving camera. International Journal of Computer Vision, 8(2), 123–151.

    Article  Google Scholar 

  • Ng, R. (2006). Digital light field photography. PhD thesis, Stanford University.

  • Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P., et al. (2005). Light field photography with a hand-held plenoptic camera. Computer Science Technical Report, 2(11), 1–11.

    Google Scholar 

  • Nistér, D. (2004). Untwisting a projective reconstruction. International Journal of Computer Vision, 60(2), 165–183.

    Article  Google Scholar 

  • Nousias, S., Lourakis, M., & Bergeles, C. (2019). Large-scale, metric structure from motion for unordered light fields. In IEEE conference on computer vision and pattern recognition (CVPR) (pp 3292–3301).

  • Paudel, D. P., Van Gool, L. (2018). Sampling algebraic varieties for robust camera autocalibration. In European conference on computer vision (ECCV) (pp. 275–292)

  • Pless, R. (2003). Using many cameras as one. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2, 587–593.

    Google Scholar 

  • Pollefeys, M., & Van Gool, L. (1999). Stratified self-calibration with the modulus constraint. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 21(8), 707–724

  • Pollefeys, M., Koch, R., & Van Gool, L. (1999). Self-calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters. International Journal of Computer Vision, 32(1), 7–25.

    Article  Google Scholar 

  • Pottmann, H., & Wallner, J. (2009). Computational line geometry. Springer Science & Business Media.

  • Raytrix. (2013). 3d light field camera technology. http://www.raytrix.de.

  • Ren, Z., Zhang, Q., Zhu, H., & Wang, Q. (2017). Extending the FOV from disparity and color consistencies in multiview light fields. In IEEE international conference on image processing (ICIP) (pp. 1157–1161).

  • Seo, Y., Heyden, A., & Cipolla, R. (2001). A linear iterative method for auto-calibration using the dac equation. In IEEE conference on computer vision and pattern recognition (CVPR).

  • Sturm, P. (2005). Multi-view geometry for general camera models. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1, 206–212.

    Google Scholar 

  • Triggs, B. (1997). Autocalibration and the absolute quadric. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 609—-614).

  • Vianello, A., Ackermann, J., Diebold, M., & Jähne, B. (2018). Robust hough transform based 3d reconstruction from circular light fields. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 7327–7335).

  • Zhang, Q., & Wang, Q. (2018). Common self-polar triangle of concentric conics for light field camera calibration. In Asian conference on computer vision (ACCV) (pp. 18–33).

  • Zhang, Q., Ling, J., Wang, Q., & Yu, J. (2019a). Ray-space projection model for light field camera. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 10121–10129).

  • Zhang, Q., Wang, X., & Wang, Q. (2019b). Light field planar homography and its application. In Optoelectronic imaging and multimedia technology VI (Vol. 11187, p. 111870S).

  • Zhang, Q., Zhang, C., Ling, J., Wang, Q., & Yu, J. (2019c). A generic multi-projection-center model and calibration method for light field cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 41(11), 2539–2552.

  • Zhang, Q., Wang, Q., Li, H., & Yu, J. (2020). Ray-space epipolar geometry for light field cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 1. https://doi.org/10.1109/TPAMI.2020.3025949.

  • Zhang, Y., Li, Z., Yang, W., Yu, P., Lin, H., & Yu, J. (2017a). The light field 3d scanner. In IEEE international conference on computational photography (ICCP) (pp. 1–9).

  • Zhang, Y., Yu, P., Yang, W., Ma, Y., & Yu, J. (2017b). Ray space features for plenoptic structure-from-motion. In IEEE international Conference on computer vision (ICCV) (pp. 4631–4639).

Download references

Acknowledgements

The work was supported by NSFC under Grant 61531014, 61801396, 62031023. We thank the editors and reviewers for valuable suggestions on contents and experiments. We also thank Ying Feng for helpful supports on data collection. Qi Zhang was also supported by Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University under CX201919 and China Scholarship Council (CSC).

Author information

Authors and Affiliations

  1. School of Computer Science, Northwestern Polytechnical University, Xi’an, 710072, China

    Qi Zhang, Xue Wang & Qing Wang

  2. ANU and ACRV, Australian National University, Canberra, ACT, 2600, Australia

    Hongdong Li

Authors
  1. Qi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongdong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xue Wang or Qing Wang.

Additional information

Communicated by Adrien Bartoli.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work was supported by NSFC under Grant 61531014, 61801396, 62031023.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Q., Li, H., Wang, X. et al. 3D Scene Reconstruction with an Un-calibrated Light Field Camera. Int J Comput Vis 129, 3006–3026 (2021). https://doi.org/10.1007/s11263-021-01516-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-021-01516-1

Keywords

Search

Navigation