Seeing Deeply and Bidirectionally: A Deep Learning Approach for Single Image Reflection Removal

  • Jie YangEmail author
  • Dong Gong
  • Lingqiao Liu
  • Qinfeng Shi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11207)


Reflections often obstruct the desired scene when taking photos through glass panels. Removing unwanted reflection automatically from the photos is highly desirable. Traditional methods often impose certain priors or assumptions to target particular type(s) of reflection such as shifted double reflection, thus have difficulty to generalize to other types. Very recently a deep learning approach has been proposed. It learns a deep neural network that directly maps a reflection contaminated image to a background (target) image (i.e.reflection free image) in an end to end fashion, and outperforms the previous methods. We argue that, to remove reflection truly well, we should estimate the reflection and utilize it to estimate the background image. We propose a cascade deep neural network, which estimates both the background image and the reflection. This significantly improves reflection removal. In the cascade deep network, we use the estimated background image to estimate the reflection, and then use the estimated reflection to estimate the background image, facilitating our idea of seeing deeply and bidirectionally.


  1. 1.
    Xue, T., Rubinstein, M., Liu, C., Freeman, W.T.: A computational approach for obstruction-free photography. ACM Trans. Graph. 34(4), 79 (2015)CrossRefGoogle Scholar
  2. 2.
    Szeliski, R., Avidan, S., Anandan, P.: Layer extraction from multiple images containing reflections and transparency. In: CVPR, vol. 1, pp. 246–253. IEEE (2000)Google Scholar
  3. 3.
    Li, Y., Brown, M.S.: Single image layer separation using relative smoothness. In: CVPR, pp. 2752–2759. IEEE (2014)Google Scholar
  4. 4.
    Levin, A., Weiss, Y.: User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans. PAMI 29(9), 1647–1654 (2007)CrossRefGoogle Scholar
  5. 5.
    Li, Y., Brown, M.S.: Exploiting reflection change for automatic reflection removal. In: ICCV, pp. 2432–2439 (2013)Google Scholar
  6. 6.
    Guo, X., Cao, X., Ma, Y.: Robust separation of reflection from multiple images. In: CVPR, pp. 2187–2194. IEEE (2014)Google Scholar
  7. 7.
    Sarel, B., Irani, M.: Separating transparent layers through layer information exchange. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 328–341. Springer, Heidelberg (2004). Scholar
  8. 8.
    Han, B.J., Sim, J.Y.: Reflection removal using low-rank matrix completion. In: CVPR, vol. 2. IEEE (2017)Google Scholar
  9. 9.
    Shih, Y., Krishnan, D., Durand, F., Freeman, W.T.: Reflection removal using ghosting cues. In: CVPR, pp. 3193–3201. IEEE (2015)Google Scholar
  10. 10.
    Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.P.: A generic deep architecture for single image reflection removal and image smoothing. In: ICCV, pp. 3258–3267 (2017)Google Scholar
  11. 11.
    Arvanitopoulos, N., Achanta, R., Süsstrunk, S.: Single image reflection suppression. In: CVPR, pp. 1752–1760. IEEE (2017)Google Scholar
  12. 12.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, vol. 2, p. 4. IEEE (2017)Google Scholar
  13. 13.
    Gong, D., et al.: From motion blur to motion flow: a deep learning solution for removing heterogeneous motion blur. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  14. 14.
    Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)Google Scholar
  15. 15.
    Gong, D., Zhang, Z., Shi, Q., Hengel, A.V.D., Shen, C., Zhang, Y.: Learning an optimizer for image deconvolution. arXiv preprint arXiv:1804.03368 (2018)
  16. 16.
    Levin, A., Zomet, A., Weiss, Y.: Learning to perceive transparency from the statistics of natural scenes. In: NIPS, pp. 1271–1278 (2003)Google Scholar
  17. 17.
    Levin, A., Zomet, A., Weiss, Y.: Separating reflections from a single image using local features. In: CVPR, vol. 1, pp. 306–313. IEEE (2004)Google Scholar
  18. 18.
    Wan, R., Shi, B., Hwee, T.A., Kot, A.C.: Depth of field guided reflection removal. In: ICIP, pp. 21–25. IEEE (2016)Google Scholar
  19. 19.
    Zhang, X., Ng, R., Chen, Q.: Single image reflection separation with perceptual losses. In: CVPR. IEEE (2018)Google Scholar
  20. 20.
    Wan, R., Shi, B., Duan, L.Y., Tan, A.H., Kot, A.C.: CRRN: multi-scale guided concurrent reflection removal network. In: CVPR, pp. 4777–4785. IEEE (2018)Google Scholar
  21. 21.
    Agrawal, A., Raskar, R., Nayar, S.K., Li, Y.: Removing photography artifacts using gradient projection and flash-exposure sampling. ACM Trans. Graph. 24(3), 828–835 (2005)CrossRefGoogle Scholar
  22. 22.
    Schechner, Y.Y., Kiryati, N., Basri, R.: Separation of transparent layers using focus. IJCV 39(1), 25–39 (2000)CrossRefGoogle Scholar
  23. 23.
    Gai, K., Shi, Z., Zhang, C.: Blind separation of superimposed moving images using image statistics. IEEE Trans. PAMI 34(1), 19–32 (2012)CrossRefGoogle Scholar
  24. 24.
    Sinha, S.N., Kopf, J., Goesele, M., Scharstein, D., Szeliski, R.: Image-based rendering for scenes with reflections. ACM Trans. Graph. 31(4), 100–1 (2012)CrossRefGoogle Scholar
  25. 25.
    Yang, J., Li, H., Dai, Y., Tan, R.T.: Robust optical flow estimation of double-layer images under transparency or reflection. In: CVPR, pp. 1410–1419 (2016)Google Scholar
  26. 26.
    Schechner, Y.Y., Shamir, J., Kiryati, N.: Polarization and statistical analysis of scenes containing a semireflector. JOSA A 17(2), 276–284 (2000)CrossRefGoogle Scholar
  27. 27.
    Kong, N., Tai, Y.W., Shin, J.S.: A physically-based approach to reflection separation: from physical modeling to constrained optimization. IEEE Trans. PAMI 36(2), 209–221 (2014)CrossRefGoogle Scholar
  28. 28.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  29. 29.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR, pp. 5967–5976. IEEE (2017)Google Scholar
  30. 30.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456 (2015)Google Scholar
  31. 31.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2242–2251. IEEE (2017)Google Scholar
  32. 32.
    Lettry, L., Vanhoey, K., van Gool, L.: DARN: a deep adversarial residual network for intrinsic image decomposition. In: WACV, pp. 1359–1367. IEEE (2018)Google Scholar
  33. 33.
    Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., Webb, R.: Learning from simulated and unsupervised images through adversarial training. In: CVPR, vol. 2, p. 5. IEEE (2017)Google Scholar
  34. 34.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  35. 35.
    Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The Pascal visual object classes (VOC) challenge. IJCV 88(2), 303–338 (2010)CrossRefGoogle Scholar
  36. 36.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  37. 37.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: ICCV, pp. 1026–1034 (2015)Google Scholar
  38. 38.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  39. 39.
    Wan, R., Shi, B., Duan, L.Y., Tan, A.H., Kot, A.C.: Benchmarking single-image reflection removal algorithms. In: ICCV, pp. 3942–3950. IEEE (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Jie Yang
    • 1
    Email author
  • Dong Gong
    • 1
  • Lingqiao Liu
    • 1
  • Qinfeng Shi
    • 1
  1. 1.School of Computer ScienceUniversity of AdelaideAdelaideAustralia

Personalised recommendations