Skip to main content
SpringerLink
Log in

Face Image Reflection Removal

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Face images captured through glass are usually contaminated by reflections. The low-transmitted reflections make the reflection removal more challenging than for general scenes because important facial features would be completely occluded. In this paper, we propose and solve the face image reflection removal problem. We recover the important facial structures by incorporating inpainting ideas into a guided reflection removal framework, which takes two images as the input and considers various face-specific priors. We use a newly collected face reflection image dataset to train our model and compare with state-of-the-art methods. The proposed method shows advantages in estimating reflection-free face images for improving face recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Amos, B., Ludwiczuk, B., & Satyanarayanan, M. (2016). Openface: A general-purpose face recognition library with mobile applications. Pittsburgh: CMU School of Computer Science.

    Google Scholar 

  • Arvanitopoulos, N., Achanta, R., & Susstrunk, S. (2017). Single image reflection suppression. In Proceedings of computer vision and pattern recognition (CVPR).

  • Baker, S., & Kanade, T. (2000). Hallucinating faces. In FG.

  • Chang, Y., Jung, C., Sun, J., et al. Siamese dense network for reflection removal with flash and no-flash image pairs. International Journal of Computer Vision, 128, 1673–1698. https://doi.org/10.1007/s11263-019-01276-z.

  • Chen, Y., Tai, Y., Liu, X., Shen, C., & Yang, J. (2018). FSRNet: End-to-end learning face super-resolution with facial priors. In Proceedings of computer vision and pattern recognition (CVPR).

  • Deng, J., Cheng, S., Xue, N., Zhou, Y., & Zafeiriou, S. (2017). UV-GAN: Adversarial facial uv map completion for pose-invariant face recognition. arXiv preprintarXiv:1712.04695.

  • Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010). The Pascal visual object classes (VOC) challenge. Springer International Journal of Computer Vision, 88(2), 303–338.

    Article  Google Scholar 

  • Fan, Q., Yang, J., Hua, G., Chen, B., & Wipf, D. (2017). A generic deep architecture for single image reflection removal and image smoothing. In Proceedings of International Conference on Computer Vision (ICCV).

  • Fan, Q., Yin, Y., Chen, D., Wang, Y., Aviles-Rivero, A., Li, R., Schnlieb, C.-B., Lischinski, D., & Chen, B. (2019). Deep reflection prior. arXiv preprintarXiv:1912.03623.

  • Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazırbaş, C., Golkov, V., Van der Smagt, P., Cremers, D., & Brox, T. (2015). Flownet: Learning optical flow with convolutional networks. arXiv preprintarXiv:1504.06852.

  • Gai, K., Shi, Z., & Zhang, C. (2012). Blind separation of superimposed moving images using image statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34, 19–32.

    Article  Google Scholar 

  • Gai, K., Shi, Z., & Zhang, C. (2014). Robust separation of reflection from multiple images. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., & Smola, A. J. (2007). A kernel method for the two-sample-problem. In Proceedings of neural information processing systems (NeurIPS).

  • Han, B.-J., & Sim, J.-Y. (2018). Glass reflection removal using co-saliency-based image alignment and low-rank matrix completion in gradient domain. IEEE Transactions on Image Processing, 27, 4873–4888.

    Article  MathSciNet  Google Scholar 

  • Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprintarXiv:1704.04861.

  • Jaderberg, M., Simonyan, K., Zisserman, A., et al. (2015). Spatial transformer networks. In Proceedings of neural information processing systems (NeurIPS).

  • Jason, J. Y., Harley, A. W., & Derpanis, K. G. (2016). Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In Proceedings of European Conference on Computer Vision (ECCV).

  • Le, V., Brandt, J., Lin, Z., Bourdev, L., & Huang, T. S. (2012). Interactive facial feature localization. In Proceedings of European Conference on Computer Vision (ECCV).

  • Learned-Miller, E., Huang, G. B., RoyChowdhury, A., Li, H., & Hua, G. (2016). Labeled faces in the wild: A survey. In Proceedings of neural information processing systems (NeurIPS).

  • Lee, D., Yang, M.-H., & Oh, S. (2018). Generative single image reflection separation. arXiv preprintarXiv:1801.04102.

  • Li, H., Pan, S. J., Wang, S., & Kot, A. C. (2018a). Domain generalization with adversarial feature learning. In Proceedings of computer vision and pattern recognition (CVPR).

  • Li, P., Xie, J., Wang, Q., & Zuo, W. (2017a). Is second-order information helpful for large-scale visual recognition. In Proceedings of International Conference on Computer Vision (ICCV).

  • Li, X., Liu, M., Ye, Y., Zuo, W., Lin, L., & Yang, R. (2018b). Learning warped guidance for blind face restoration. In Proceedings of European Conference on Computer Vision (ECCV).

  • Li, Y., & Brown, M. (2013). Exploiting reflection change for automatic reflection removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Li, Y., & Brown, M. S. (2014). Single image layer separation using relative smoothness. In Proceedings of computer vision and pattern recognition (CVPR).

  • Li, Y., Liu, S., Yang, J., & Yang, M.-H. (2017b). Generative face completion. In Proceedings of computer vision and pattern recognition (CVPR).

  • Li, Y., Swersky, K., & Zemel, R. (2015). Generative moment matching networks. In International Conference on Machine Learning.

  • Lin, D., & Tang, X. (2007). Quality-driven face occlusion detection and recovery. In Proceedings of computer vision and pattern recognition (CVPR).

  • Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., et al. (2014). Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision (ECCV).

  • Liu, C., Shum, H.-Y., & Freeman, W. T. (2007). Face hallucination: Theory and practice. Springer International Journal of Computer Vision, 75(1), 115–134.

    Article  Google Scholar 

  • Liu, Z., Luo, P., Wang, X., & Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV).

  • Lyu, Y., Cui, Z., Li, S., Pollefeys, M., & Shi, B. (2019). Reflection separation using a pair of unpolarized and polarized images. In Advances in neural information processing systems.

  • Ma, D., Wan, R., Shi, B., Kot, A. C., & Duan, L.-Y. (2019). Learning to jointly generate and separate reflections. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

  • Pan, J., Hu, Z., Su, Z., & Yang, M.-H. (2014). Deblurring face images with exemplars. In Proceedings of European Conference on Computer Vision (ECCV).

  • Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context encoders: Feature learning by inpainting. In Proceedings of computer vision and pattern recognition (CVPR).

  • Pitie, F., Kokaram, A. C., & Dahyot, R. (2005). N-dimensional probability density function transfer and its application to color transfer. In International Conference on Computer Vision (ICCV).

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention.

  • Shen, Z., Lai, W.-S., Xu, T., Kautz, J., & Yang, M.-H. (2018). Deep semantic face deblurring. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

  • Shih, Y., Krishnan, D., Durand, F., & Freeman, W. T. (2015). Reflection removal using ghosting cues. In Proceedings of computer vision and pattern recognition (CVPR).

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprintarXiv:1409.1556.

  • Smola, A., Gretton, A., Song, L., & Schölkopf, B. (2007). A Hilbert space embedding for distributions. In International Conference on Algorithmic Learning Theory.

  • Song, Y., Zhang, J., He, S., Bao, L., & Yang, Q. (2017). Learning to hallucinate face images via component generation and enhancement. arXiv preprintarXiv:1708.00223.

  • Sun, C., Liu, S., Yang, T., Zeng, B., Wang, Z., & Liu, G. (2013). Automatic reflection removal using gradient intensity and motion cues. In Proceedings of IEEE conference on Computer Vision and Pattern Recognition (CVPR).

  • Torralba, A., & Efros, A. A. (2011). Unbiased look at dataset bias. In CVPR 2011 (pp. 1521–1528). IEEE.

  • Wan, R., Shi, B., Duan, L.-Y., Tan, A.-H., & Kot, A. C. (2017a). Benchmarking single-image reflection removal algorithms. In Proceedings of International Conference on Computer Vision (ICCV).

  • Wan, R., Shi, B., Duan, L.-Y., Tan, A.-H., & Kot, A. C. (2018a). CRRN: Multi-scale guided concurrent reflection removal network. In Proceedings of computer vision and pattern recognition (CVPR).

  • Wan, R., Shi, B., Duan, L.-Y., Tan, A.-H., Gao, W., & Kot, A. C. (2018b). Region-aware reflection removal with unified content and gradient priors. IEEE Transactions on Image Processing, 27(6), 2927–2941.

    Article  MathSciNet  Google Scholar 

  • Wan, R., Shi, B., Li, H., Duan, L.-Y., Tan, A.-H., & Chichung, A. K. (2019). CoRRN: Cooperative reflection removal network. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2019.2921574.

  • Wan, R., Shi, B., Tan, A. H., & Kot, A. C. (2016). Depth of field guided reflection removal. In Proceedings of International Conference on Image Processing (ICIPa).

  • Wan, R., Shi, B., Tan, A., & Kot, A. C. (2017b). Sparsity based reflection removal using external patch search. In Proceedings of International Conference on Multimedia and Expo.

  • Wang, X., & Tang, X. (2005). Hallucinating face by eigentransformation. In IEEE transactions on systems, man, and cybernetics, part C (applications and reviews) (Vol. 35, no. 3, pp. 425–434). https://doi.org/10.1109/TSMCC.2005.848171.

  • Wei, K., Yang, J., Fu, Y., Wipf, D., & Huang, H. (2019). Single image reflection removal exploiting misaligned training data and network enhancements. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Wu, X., He, R., Sun, Z., & Tan, T. (2018). A light CNN for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security, 13(11), 2884–2896.

    Article  Google Scholar 

  • Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2017). Video enhancement with task-oriented flow. arXiv preprintarXiv:1711.09078.

  • Yang, C.-Y., Liu, S., & Yang, M.-H. (2013). Structured face hallucination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

  • Yang, J., Gong, D., Liu, L., & Shi, Q. (2018). Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In Proceedings of European Conference on Computer Vision (ECCV).

  • Yang, J., Li, H., Dai, Y., & Tan, R. T. (2016). Robust optical flow estimation of double-layer images under transparency or reflection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Yi, D., Lei, Z., Liao, S., & Li, S. Z. (2014). Learning face representation from scratch. arXiv preprintarXiv:1411.7923.

  • Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letter, 23(10), 1499–1503.

    Article  Google Scholar 

  • Zhang, S., He, R., Sun, Z., & Tan, T. (2018a). Demeshnet: Blind face inpainting for deep meshface verification. IEEE Transactions on Information Forensics and Security, 13(3), 637–647.

    Article  Google Scholar 

  • Zhang, X., Ng, R., & Chen, Q. (2018b). Single image reflection separation with perceptual losses. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, Singapore

    Renjie Wan, Haoliang Li & Alex C. Kot

  2. National Engineering Laboratory for Video Technology, Department of CS, Peking University, Beijing, China

    Boxin Shi & Ling-Yu Duan

  3. Peng Cheng Laboratory, Shenzhen, China

    Boxin Shi & Ling-Yu Duan

Authors
  1. Renjie Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Boxin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haoliang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ling-Yu Duan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alex C. Kot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Renjie Wan or Boxin Shi.

Additional information

Communicated by Yoichi Sato.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The research work was done at the Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University. The work is supported in part by the Wallenberg-NTU Presidential Postdoctoral Fellowship, the NTU-PKU Joint Research Institute, a collaboration between the Nanyang Technological University and Peking University that is sponsored by a donation from the Ng Teng Fong Charitable Foundation, and the Science and Technology Foundation of Guangzhou Huangpu Development District under Grant 201902010028. This research is in part supported by the National Natural Science Foundation of China under Grants 61872012 and U1611461, and Beijing Academy of Artificial Intelligence (BAAI).

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2388 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wan, R., Shi, B., Li, H. et al. Face Image Reflection Removal. Int J Comput Vis 129, 385–399 (2021). https://doi.org/10.1007/s11263-020-01372-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-020-01372-5

Keywords

Search

Navigation