Reflection Separation via Multi-bounce Polarization State Tracing

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)


Reflection removal from photographs is an important task in computational photography, but also for computer vision tasks that involve imaging through windows and similar settings. Traditionally, the problem is approached as a single reflection removal problem under very controlled scenarios. In this paper we aim to generalize the reflection removal to real-world scenarios with more complicated light interactions. To this end, we propose a simple yet efficient learning framework for supervised image reflection separation with a polarization-guided ray-tracing model and loss function design. Instead of a conventional image sensor, we use a polarization sensor that instantaneously captures four linearly polarized photos of the scene in the same image. Through a combination of a new polarization-guided image formation model and a novel supervised learning framework for the interpretation of a ray-tracing image formation model, a general method is obtained to tackle general image reflection removal problems. We demonstrate our method with extensive experiments on both real and synthetic data and demonstrate the unprecedented quality of image reconstructions.


Reflection removal Polarization simulation engine Ray-tracing Polarization tracing 

Supplementary material

504454_1_En_46_MOESM1_ESM.pdf (31.7 mb)
Supplementary material 1 (pdf 32479 KB)


  1. 1.
    Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: A generic deep architecture for single image reflection removal and image smoothing. In: CVPR (2017)Google Scholar
  2. 2.
    Fan, Q., Yang, J., Wipf, D., Chen, B., Tong, X.: Image smoothing via unsupervised learning. TOG 37, 1–4 (2019)CrossRefGoogle Scholar
  3. 3.
    Gandelsman, Y., Shocher, A., Irani, M.: Double-dip: unsupervised image decomposition via coupled deep-image-priors. In: CVPR (2019)Google Scholar
  4. 4.
    Zhang, X., Ng, R., Chen, Q.: Single image reflection separation with perceptual losses. In: CVPR (2018)Google Scholar
  5. 5.
    Levin, A., Weiss, Y.: User assisted separation of reflections from a single image using a sparsity prior. PAMI 29, 1647–1654 (2007)CrossRefGoogle Scholar
  6. 6.
    Li, Y., Brown, M.S.: Exploiting reflection change for automatic reflection removal. In: ICCV (2013)Google Scholar
  7. 7.
    Levin, A., Zomet, A., Weiss, Y.: Learning to perceive transparency from the statistics of natural scenes. In: NeurIPS (2003)Google Scholar
  8. 8.
    Levin, A., Zomet, A., Weiss, Y.: Separating reflections from a single image using local features. In: CVPR (2004)Google Scholar
  9. 9.
    Li, Y., Brown, M.S.: Single image layer separation using relative smoothness. In: CVPR (2014)Google Scholar
  10. 10.
    Shih, Y., Krishnan, D., Durand, F., Freeman, W.T.: Reflection removal using ghosting cues. In: CVPR (2015)Google Scholar
  11. 11.
    Arvanitopoulos, N., Achanta, R., Susstrunk, S.: Single image reflection suppression. In: CVPR (2017)Google Scholar
  12. 12.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  13. 13.
    Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via L0 gradient minimization. TOG 30, 11 (2011)Google Scholar
  14. 14.
    Wan, R., Shi, B., Hwee, T.A., Kot, A.C.: Depth of field guided reflection removal. In: ICIP(2016)Google Scholar
  15. 15.
    Yang, J., Gong, D., Liu, L., Shi, Q.: Seeing deeply and bidirectionally: a deep learning approach for single image reflection removal. In: ECCV (2018)Google Scholar
  16. 16.
    Jin, M., Süsstrunk, S., Favaro, P.: Learning to see through reflections. In: ICCP (2018)Google Scholar
  17. 17.
    Wan, R., Shi, B., Duan, L.Y., Tan, A.H., Kot, A.C.: CRRN: multi-scale guided concurrent reflection removal network. In: CVPR (2018)Google Scholar
  18. 18.
    Wen, Q., Tan, Y., Qin, J., Liu, W., Han, G., He, S.: Single image reflection removal beyond linearity. In: CVPR (2019)Google Scholar
  19. 19.
    Wei, K., Yang, J., Fu, Y., David, W., Huang, H.: Single image reflection removal exploiting misaligned training data and network enhancements. In: CVPR (2019)Google Scholar
  20. 20.
    Xue, T., Rubinstein, M., Liu, C., Freeman, W.T.: A computational approach for obstruction-free photography. TOG 34, 1–11 (2015)CrossRefGoogle Scholar
  21. 21.
    Guo, X., Cao, X., Ma, Y.: Robust separation of reflection from multiple images. In: CVPR (2014)Google Scholar
  22. 22.
    Han, B.J., Sim, J.Y.: Reflection removal using low-rank matrix completion. In: CVPR (2017)Google Scholar
  23. 23.
    Lyu, Y., Cui, Z., Li, S., Pollefeys, M., Shi, B.: Reflection separation using a pair of unpolarized and polarized images. In: NeurIPS (2019)Google Scholar
  24. 24.
    Wieschollek, P., Gallo, O., Gu, J., Kautz, J.: Separating reflection and transmission images in the wild. In: ECCV (2018)Google Scholar
  25. 25.
    Kong, N., Tai, Y., Shin, J.S.: A physically-based approach to reflection separation: from physical modeling to constrained optimization. PAMI 36, 209–221 (2014)CrossRefGoogle Scholar
  26. 26.
    Farid, H., Adelson, E.H.: Separating reflections and lighting using independent components analysis. In: CVPR (1999)Google Scholar
  27. 27.
    Miyazaki, D., Ikeuchi, K.: Inverse polarization raytracing: estimating surface shapes of transparent objects. In: CVPR (2005)Google Scholar
  28. 28.
    Qiu, S., Fu, Q., Wang, C., Heidrich, W.: Polarization demosaicking for monochrome and color polarization focal plane arrays. In: Vision, Modeling and Visualization (2019)Google Scholar
  29. 29.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)Google Scholar
  30. 30.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.King Abdullah University of Science and TechnologyThuwalSaudi Arabia

Personalised recommendations