Fast Light Field Reconstruction with Deep Coarse-to-Fine Modeling of Spatial-Angular Clues

  • Henry Wing Fung Yeung
  • Junhui HouEmail author
  • Jie Chen
  • Yuk Ying Chung
  • Xiaoming Chen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11210)


Densely-sampled light fields (LFs) are beneficial to many applications such as depth inference and post-capture refocusing. However, it is costly and challenging to capture them. In this paper, we propose a learning based algorithm to reconstruct a densely-sampled LF fast and accurately from a sparsely-sampled LF in one forward pass. Our method uses computationally efficient convolutions to deeply characterize the high dimensional spatial-angular clues in a coarse-to-fine manner. Specifically, our end-to-end model first synthesizes a set of intermediate novel sub-aperture images (SAIs) by exploring the coarse characteristics of the sparsely-sampled LF input with spatial-angular alternating convolutions. Then, the synthesized intermediate novel SAIs are efficiently refined by further recovering the fine relations from all SAIs via guided residual learning and stride-2 4-D convolutions. Experimental results on extensive real-world and synthetic LF images show that our model can provide more than 3 dB advantage in reconstruction quality in average than the state-of-the-art methods while being computationally faster by a factor of 30. Besides, more accurate depth can be inferred from the reconstructed densely-sampled LFs by our method.


Light field Deep learning Convolutional neural network Super resolution View synthesis 



This work was supported in part by the CityU Start-up Grant for New Faculty under Grant 7200537/CS and in part by the Hong Kong RGC Early Career Scheme Funds 9048123 (CityU 21211518).


  1. 1.
    Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 606–619 (2014)CrossRefGoogle Scholar
  2. 2.
    Jeon, H.G., et al.: Accurate depth map estimation from a lenslet light field camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1547–1555 (2015)Google Scholar
  3. 3.
    Wang, T.C., Efros, A.A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3487–3495 (2015)Google Scholar
  4. 4.
    Chen, J., Hou, J., Ni, Y., Chau, L.P.: Accurate light field depth estimation with superpixel regularization over partially occluded regions. IEEE Trans. Image Process. 27(10), 4889–4900 (2018)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Fiss, J., Curless, B., Szeliski, R.: Refocusing plenoptic images using depth-adaptive splatting. In: Proceedings of IEEE International Conference on Computational Photography, pp. 1–9 (2014)Google Scholar
  6. 6.
    Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 31–42. ACM (1996)Google Scholar
  7. 7.
    Jones, A., McDowall, I., Yamada, H., Bolas, M., Debevec, P.: Rendering for an interactive 360 light field display. ACM Trans. Graph. 26(3), 40 (2007)CrossRefGoogle Scholar
  8. 8.
    Ihrke, I., Restrepo, J., Mignard-Debise, L.: Principles of light field imaging: briefly revisiting 25 years of research. IEEE Sig. Process. Mag. 33(5), 59–69 (2016)CrossRefGoogle Scholar
  9. 9.
    Wu, G., et al.: Light field image processing: an overview. IEEE J. Sel. Top. Sig. Process. 11(7), 926–954 (2017)CrossRefGoogle Scholar
  10. 10.
    Wilburn, B., et al.: High performance imaging using large camera arrays. ACM Trans. Graph. 24(3), 765–776 (2005)MathSciNetCrossRefGoogle Scholar
  11. 11.
  12. 12.
  13. 13.
    Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. 35(6), 193 (2016)CrossRefGoogle Scholar
  14. 14.
    Wu, G., Zhao, M., Wang, L., Dai, Q., Chai, T., Liu, Y.: Light field reconstruction using deep convolutional network on EPI. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6319–6327 (2017)Google Scholar
  15. 15.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). Scholar
  16. 16.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)Google Scholar
  17. 17.
    Lai, W.S., Huang, J.B., Ahuja, N., Yang, M.H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, no. 3, pp. 5835–5843 (2017)Google Scholar
  18. 18.
    Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2798 (2017)Google Scholar
  19. 19.
    Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., So Kweon, I.: Learning a deep convolutional network for light-field image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 24–32 (2015)Google Scholar
  20. 20.
    Yoon, Y., Jeon, H.G., Yoo, D., Lee, J.Y., Kweon, I.S.: Light-field image super-resolution using convolutional neural network. IEEE Sig. Process. Lett. 24(6), 848–852 (2017)CrossRefGoogle Scholar
  21. 21.
    Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Comput. Sci. Tech. Rep. CSTR 2(11), 1–11 (2005)Google Scholar
  22. 22.
    Rigamonti, R., Sironi, A., Lepetit, V., Fua, P.: Learning separable filters. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2754–2761 (2013)Google Scholar
  23. 23.
    Yan, L.Q., Mehta, S.U., Ramamoorthi, R., Durand, F.: Fast 4D sheared filtering for interactive rendering of distribution effects. ACM Trans. Graph. 35(1), 7 (2015)CrossRefGoogle Scholar
  24. 24.
    Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive separable convolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 261–270 (2017)Google Scholar
  25. 25.
    Wang, T.C., Zhu, J.Y., Hiroaki, E., Chandraker, M., Efros, A.A., Ramamoorthi, R.: A 4D light-field dataset and CNN architectures for material recognition. In: Proceedings of the European Conference on Computer Vision, pp. 121–138 (2016)CrossRefGoogle Scholar
  26. 26.
    Vedaldi, A., Lenc, K.: MatConvNet - convolutional neural networks for MATLAB. In: Proceedings of the ACM International Conference on Multimedia, pp. 689–692 (2015)Google Scholar
  27. 27.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)Google Scholar
  28. 28.
    Raj, A.S., Lowney, M., Shah, R., Wetzstein, G.: Stanford Lytro light field archive.
  29. 29.
    Levoy, M., Ng, R., Adams, A., Footer, M., Horowitz, M.: Light field microscopy. ACM Trans. Graph. 25(3), 924–934 (2006)CrossRefGoogle Scholar
  30. 30.
    Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: VMV, pp. 225–226. Citeseer (2013)Google Scholar
  31. 31.
    Blender Online Community: Blender - a 3D modelling and rendering package. Blender Foundation, Blender Institute, AmsterdamGoogle Scholar
  32. 32.
    Rerabek, M., Ebrahimi, T.: New light field image dataset. In: Proceedings of the 8th International Conference on Quality of Multimedia Experience. Number EPFL-CONF-218363 (2016)Google Scholar
  33. 33.
    Hou, J., Chen, J., Chau, L.P.: Light field image compression based on bi-level view compensation with rate-distortion optimization. IEEE Trans. Circ. Syst. Video Technol. 1 (2018)Google Scholar
  34. 34.
    Chen, J., Hou, J., Chau, L.P.: Light field compression with disparity-guided sparse coding based on structural key views. IEEE Trans. Image Process. 27(1), 314–324 (2018)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Henry Wing Fung Yeung
    • 1
  • Junhui Hou
    • 2
    Email author
  • Jie Chen
    • 3
  • Yuk Ying Chung
    • 1
  • Xiaoming Chen
    • 4
  1. 1.School of Information TechnologiesUniversity of SydneySydneyAustralia
  2. 2.Department of Computer ScienceCity University of Hong KongKowloonHong Kong
  3. 3.School of Electrical and Electronics EngineeringNanyang Technological UniversitySingaporeSingapore
  4. 4.School of Information Science and TechnologyUniversity of Science and Technology of ChinaHefeiChina

Personalised recommendations