Abstract
In this paper, we address the problem of reflection removal and deblurring from a single image captured by a plenoptic camera. We develop a two-stage approach to recover the scene depth and high resolution textures of the reflected and transmitted layers. For depth estimation in the presence of reflections, we train a classifier through convolutional neural networks. For recovering high resolution textures, we assume that the scene is composed of planar regions and perform the reconstruction of each layer by using an explicit form of the plenoptic camera point spread function. The proposed framework also recovers the sharp scene texture with different motion blurs applied to each layer. We demonstrate our method on challenging real and synthetic images.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Guo, X., Cao, X., Ma, Y.: Robust separation of reflection from multiple images. In: CVPR, pp. 2195–2202 (2014)
Xue, T., Rubinstein, M., Liu, C., Freeman, W.T.: A computational approach for obstruction-free photography. ACM Trans. Graph. 34, 7901–7911 (2015)
Li, Y., Brown, M.: Exploiting reflection change for automatic reflection removal. In: ICCV, pp. 2432–2439 (2013)
Schechner, Y.Y., Kiryati, N., Basri, R.: Separation of transparent layers using focus. Int. J. Comput. Vis. 39, 25–39 (2000)
Agrawal, A., Raskar, R., Nayar, S.K., Li, Y.: Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph. (TOG) 24, 828–835 (2005). ACM
Kong, N., Tai, Y.W., Shin, J.S.: A physically-based approach to reflection separation: from physical modeling to constrained optimization. IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014)
Levin, A., Zomet, A., Weiss, Y.: Separating reflections from a single image using local features. In: CVPR (2004)
Li, Y., Brown, M.: Single image layer separation using relative smoothness. In: CVPR, pp. 2752–2759 (2014)
Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Comput. Sci. Tech. Rep. CSTR 2(11), 1–11 (2005)
Lytro. https://www.lytro.com/
Raytrix. (http://www.raytrix.de/)
Wanner, S., Goldlücke, B.: Reconstructing reflective and transparent surfaces from epipolar plane images. In: GCPR (2013)
Wang, Q., Lin, H., Ma, Y., Kang, S.B., Yu, J.: Automatic layer separation using light field imaging. arXiv preprint arXiv:1506.04721 (2015)
Johannsen, O., Sulc, A., Goldluecke, B.: Variational separation of light field layers. In: Vision, Modelling and Visualization (VMV) (2015)
Levin, A., Weiss, Y.: User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans. Pattern Anal. Mach. Intell. 29, 1647–1654 (2007)
Szeliski, R., Avidan, S., Anandan, P.: Layer extraction from multiple images containing reflections and transparency. In: CVPR, vol. 1, pp. 246–253. IEEE (2000)
Tsin, Y., Kang, S.B., Szeliski, R.: Stereo matching with linear superposition of layers. IEEE Trans. Pattern Anal. Mach. Intell. 28, 290–301 (2006)
Shih, Y., Krishnan, D., Durand, F., Freeman, W.T.: Reflection removal using ghosting cues. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3193–3201. IEEE (2015)
Bishop, T., Favaro, P.: The light field camera: extended depth of field, aliasing and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. 34, 972–986 (2012)
Lumsdaine, A., Georgiev., T.: Full resolution lightfield rendering. Technical report, Adobe Systems (2008)
Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Patt. Anal. Mach. Intell. 36, 606–619 (2014)
Cho, D., Lee, M., Kim, S., Tai, Y.W.: Modeling the calibration pipeline of the lytro camera for high quality light-field image reconstruction. In: Proceedings ICCV (2013)
Dansereau, D.G., Pizarro, O., Williams, S.B.: Decoding, calibration and rectification for lenselet-based plenoptic cameras. In: Proceedings CVPR (2013)
Bok, Y., Jeon, H.G., Kweon, I.S.: Geometric calibration of micro-lens-based light-field cameras using line features. In: Proceedings ECCV (2014)
Tao, M., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: Proceedings ICCV (2013)
Sabater, N., Seifi, M., Drazic, V., Sandri, G., Perez, P.: Accurate disparity estimation for plenoptic images. In: Proceedings ECCV Workshops (2014)
Yu, Z., Guo, X., Ling, H., Lumsdaine, A., Yu, J.: Line assisted light field triangulation and stereo matching. In: ICCV. IEEE (2013)
Jeon, H.G., Park, J., Choe, G., Park, J., Bok, Y., Tai, Y.W., Kweon, I.S.: Accurate depth map estimation from a lenslet light field camera. In: CVPR (2015)
Wang, T.C., Efros, A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2015)
Tao, M., Su, J., Wang, T., Malik, J., Ramamoorthi, R.: Depth estimation and specular removal for glossy surfaces using point and line consistency with light-field cameras. IEEE Trans. Pattern Anal. Mach. Intell. 38(6), 1155–1169 (2015)
Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. ACM Trans. Graph. 25, 787–794 (2006)
Levin, A., Weiss, Y., Durand, F., Freeman, W.: Efficient marginal likelihood optimization in blind deconvolution. In: CVPR, pp. 2657–2664 (2011)
Cho, S., Lee, S.: Fast motion deblurring. ACM Trans. Graph. 28, 1–8 (2009)
Xu, L., Jia, J.: Two-phase kernel estimation for robust motion deblurring. In: ECCV (2010)
Xu, L., Zheng, S., Jia, J.: Unnatural L0 sparse representation for natural image deblurring. In: CVPR (2013)
Perrone, D., Favaro, P.: Total variation blind deconvolution: the devil is in the details. In: CVPR (2014)
Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. In: Proceedings CVPR (2010)
Gupta, A., Joshi, N., Zitnick, L., Cohen, M., Curless, B.: Single image deblurring using motion density functions. In: Proceedings ECCV (2010)
Hirsch, M., Schuler, C.J., Harmeling, S., Scholkopf, B.: Fast removal of non-uniform camera shake. In: Proceedings ICCV (2011)
Hu, Z., Xu, L., Yang, M.H.: Joint depth estimation and camera shake removal from single blurry image. In: CVPR (2014)
Paramanand, C., Rajagopalan, A.N.: Non-uniform motion deblurring for bilayer scenes. In: CVPR (2013)
Sorel, M., Flusser, J.: Space-variant restoration of images degraded by camera motion blur. Trans. Img. Proc. 17, 105–116 (2008)
Kim, T.H., Ahn, B., Lee, K.M.: Dynamic scene deblurring. In: The IEEE International Conference on Computer Vision (ICCV) (2013)
Whyte, O., Sivic, J., Zisserman, A.: Deblurring shaken and partially saturated images. Int. J. Comput. Vis. 110, 185–201 (2014)
Broxton, M., Grosenick, L., Yang, S., Cohen, N., Andalman, A., Deisseroth, K., Levoy, M.: Wave optics theory and 3-D deconvolution for the light field microscope. Opt. Express 21, 25418–25439 (2013)
Liang, C.K., Ramamoorthi, R.: A light transport framework for lenslet light field cameras. ACM Trans. Graph. 34, 16:1–16:19 (2015)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 448–456. ACM (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Chandramouli, P., Noroozi, M., Favaro, P. (2017). ConvNet-Based Depth Estimation, Reflection Separation and Deblurring of Plenoptic Images. In: Lai, SH., Lepetit, V., Nishino, K., Sato, Y. (eds) Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science(), vol 10113. Springer, Cham. https://doi.org/10.1007/978-3-319-54187-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-54187-7_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-54186-0
Online ISBN: 978-3-319-54187-7
eBook Packages: Computer ScienceComputer Science (R0)