Advertisement

Separating Reflection and Transmission Images in the Wild

  • Patrick WieschollekEmail author
  • Orazio Gallo
  • Jinwei Gu
  • Jan Kautz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11217)

Abstract

The reflections caused by common semi-reflectors, such as glass windows, can impact the performance of computer vision algorithms. State-of-the-art methods can remove reflections on synthetic data and in controlled scenarios. However, they are based on strong assumptions and do not generalize well to real-world images. Contrary to a common misconception, real-world images are challenging even when polarization information is used. We present a deep learning approach to separate the reflected and the transmitted components of the recorded irradiance, which explicitly uses the polarization properties of light. To train it, we introduce an accurate synthetic data generation pipeline, which simulates realistic reflections, including those generated by curved and non-ideal surfaces, non-static scenes, and high-dynamic-range scenes.

Notes

Acknowledgments

We thank the reviewers for their feedback, in particular the reviewer who suggested the experiment in Fig. 7, Hendrik P.A. Lensch for the fruitful discussions, and the people who donated half hour of their lives to take our survey.

Supplementary material

474201_1_En_6_MOESM1_ESM.pdf (94 kb)
Supplementary material 1 (pdf 93 KB)

References

  1. 1.
  2. 2.
    Schechner, Y.Y., Shamir, J., Kiryati, N.: Polarization and statistical analysis of scenes containing a semireflector. J. Opt. Soc. Am. 17(2), 276–284 (2000)CrossRefGoogle Scholar
  3. 3.
    Kong, N., Tai, Y.W., Shin, J.S.: A physically-based approach to reflection separation: from physical modeling to constrained optimization. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(2), 209–221 (2014)CrossRefGoogle Scholar
  4. 4.
    Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: A generic deep architecture for single image reflection removal and image smoothing. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3258–3267 (2017)Google Scholar
  5. 5.
    Collett, E.: Field Guide to Polarization. SPIE Press, Bellingham (2005)CrossRefGoogle Scholar
  6. 6.
    Levin, A., Weiss, Y.: User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI), 29(9) (2007)CrossRefGoogle Scholar
  7. 7.
    Li, Y., Brown, M.S.: Single image layer separation using relative smoothness. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2752–2759 (2014)Google Scholar
  8. 8.
    Arvanitopoulos, D.N., Achanta, R., Süsstrunk, S.: Single image reflection suppression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1752–1760 (2017)Google Scholar
  9. 9.
    Diamant, Y., Schechner, Y.Y.: Overcoming visual reverberations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2008)Google Scholar
  10. 10.
    Shih, Y., Krishnan, D., Durand, F., Freeman, W.T.: Reflection removal using ghosting cues. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3193–3201 (2015)Google Scholar
  11. 11.
    Li, Y., Brown, M.S.: Exploiting reflection change for automatic reflection removal. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2432–2439 (2013)Google Scholar
  12. 12.
    Xue, T., Rubinstein, M., Liu, C., Freeman, W.T.: A computational approach for obstruction-free photography. ACM Trans. Graph. (SIGGRAPH) 34(4), 79 (2015)CrossRefGoogle Scholar
  13. 13.
    Szeliski, R., Avidan, S., Anandan, P.: Layer extraction from multiple images containing reflections and transparency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 1246 (2000)Google Scholar
  14. 14.
    Guo, X., Cao, X., Ma, Y.: Robust separation of reflection from multiple images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2187–2194 (2014)Google Scholar
  15. 15.
    Han, B.J., Sim, J.Y.: Reflection removal using low-rank matrix completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  16. 16.
    Kaftory, R., Zeevi, Y.Y.: Blind separation of time/position varying mixtures. IEEE Trans. Image Process. (TIP) 22(1), 104–118 (2013)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Farid, H., Adelson, E.H.: Separating reflections and lighting using independent components analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 262–267 (1999)Google Scholar
  18. 18.
    Barros, A.K., Yamamura, T., Ohnishi, N.: Separating virtual and real objects using independent component analysis. IEICE Trans. Inf. Syst. 84(9), 1241–1248 (2001)Google Scholar
  19. 19.
    Bronstein, A.M., Bronstein, M.M., Zibulevsky, M., Zeevi, Y.Y.: Sparse ICA for blind separation of transmitted and reflected images. Int. J. Imaging Syst. Technol. 15(1), 84–91 (2005)CrossRefGoogle Scholar
  20. 20.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  21. 21.
    Mao, X., Shen, C., Yang, Y.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems (NIPS) (2016)Google Scholar
  22. 22.
    Wieschollek, P., Schölkopf, M.H.B., Lensch, H.P.A.: Learning blind motion deblurring. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  23. 23.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation (2015). arXiv preprint arXiv:1505.04597Google Scholar
  24. 24.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  25. 25.
    Wan, R., Shi, B., Duan, L.Y., Tan, A.H., Kot, A.C.: Benchmarking single-image reflection removal algorithms. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  26. 26.
    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) (2017)Google Scholar
  27. 27.
  28. 28.
  29. 29.
  30. 30.
    Gallo, O., Sen, P.: Stack-based algorithms for HDR capture and reconstruction. High Dynamic Range Video. Elsevier, Oxford (2016)CrossRefGoogle Scholar
  31. 31.
    Wieschollek, P., Schölkopf, B., Lensch, H.P.A., Hirsch, M.: End-to-end learning for image burst deblurring. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (2016)Google Scholar
  32. 32.
    Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  33. 33.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge (VOC 2012) Results (2012). http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html
  34. 34.
    Schechner, Y.Y., Kiryati, N., Shamir, J.: Separation of transparent layers by polarization analysis. In: Proceeding of the Scandinavian Conference on Image Analysis (1999)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Patrick Wieschollek
    • 1
    • 2
    Email author
  • Orazio Gallo
    • 1
  • Jinwei Gu
    • 1
  • Jan Kautz
    • 1
  1. 1.NVIDIASanta ClaraUSA
  2. 2.University of TübingenTübingenGermany

Personalised recommendations