Skip to main content
SpringerLink
Log in

A two-stage progressive shadow removal network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Removing image shadows has been a challenging task in computer vision due to its diversity and complexity. Shadow removal techniques have been greatly enhanced by deep learning and shadow image datasets, but state-of-the-art methods generally consider the information of the shadow and its neighborhood, ignoring the correlation of the features between the shadow and non-shadow regions. It leads to the resulting image presenting poor overall consistency and unnatural boundary between the original shadow and non-shadow areas. To obtain a consistent and natural shadow removal result, a two-stage progressive shadow removal network is proposed. The first stage performs a multi-exposure fusion network (MEFN) to roughly recover the shadow region features, while in the second stage, a fine-recovery network (FRN) is performed to extract the correlation among the global image contexts, accompanied by a detail feature fusion step. This coarse-to-fine process improves the overall effect of shadow removal, in terms of image quality and boundary consistency. Extensive experiments on the widely used ISTD, ISTD+ and SRD datasets show that the proposed shadow removal network outperforms most of the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from the corresponding author, [X. Chen], upon reasonable request.

References

  1. Sanches SR, Oliveira C, Sementille AC et al (2019) Challenging situations for background subtraction algorithms. Appl Intell 49:1771–1784

    Article  Google Scholar 

  2. Zou Q, Jiang H, Dai Q et al (2019) Robust lane detection from continuous driving scenes using deep neural networks. IEEE Trans Veh Technol 69(1):41–54

    Article  Google Scholar 

  3. Yuan X, Li D, Mohapatra D et al (2018) Automatic removal of complex shadows from indoor videos using transfer learning and dynamic thresholding. Comput Electr Eng 70:813–825

    Article  Google Scholar 

  4. Yu X, Tian X (2022) A fault detection algorithm for pipeline insulation layer based on immune neural network. Int J Press Vessel Pip 196(104):611

    Google Scholar 

  5. Yu X, Ye X, Zhang S (2022) Floating pollutant image target extraction algorithm based on immune extremum region. Digit Signal Proc 123(103):442

    Google Scholar 

  6. Zhang W, Zhao X, Morvan JM et al (2018) Improving shadow suppression for illumination robust face recognition. IEEE Trans Pattern Anal Mach Intell 41(3):611–624

    Article  Google Scholar 

  7. Guo R, Dai Q, Hoiem D (2012) Paired regions for shadow detection and removal. IEEE Trans Pattern Anal Mach Intell 35(12):2956–2967

    Article  Google Scholar 

  8. Yang Q, Tan KH, Ahuja N (2012) Shadow removal using bilateral filtering. IEEE Trans Image Process 21(10):4361–4368

    Article  MathSciNet  MATH  Google Scholar 

  9. Wang J, Li X, Yang J (2018) Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1788–1797

  10. Hu X, Jiang Y, Fu CW et al (2019) Mask-shadowgan: Learning to remove shadows from unpaired data. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2472–2481

  11. Jin Y, Sharma A, Tan RT (2021) Dc-shadownet: Single-image hard and soft shadow removal using unsupervised domain-classifier guided network. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 5027–5036

  12. Choi J, Yoo YJ, Choi JY (2010) Adaptive shadow estimator for removing shadow of moving object. Comp Vision Image Underst 114(9):1017–1029

    Article  Google Scholar 

  13. Si X, Feng J, Zhou J et al (2015) Detection and rectification of distorted fingerprints. IEEE Trans Pattern Anal Mach Intell 37(3):555–568

    Article  Google Scholar 

  14. Zhang L, Zhang Q, Xiao C (2015) Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Trans Image Process 24(11):4623–4636

    Article  MathSciNet  MATH  Google Scholar 

  15. Finlayson GD, Hordley SD, Lu C et al (2005) On the removal of shadows from images. IEEE Trans Pattern Anal Mach Intell 28(1):59–68

    Article  Google Scholar 

  16. Wu TP, Tang CK, Brown MS et al (2007) Natural shadow matting. ACM Transactions on Graphics (TOG) 26(2):8–es

  17. Gryka M, Terry M, Brostow GJ (2015) Learning to remove soft shadows. ACM Trans Graph (TOG) 34(5):1–15

    Article  Google Scholar 

  18. Khan SH, Bennamoun M, Sohel F et al (2015) Automatic shadow detection and removal from a single image. IEEE Trans Pattern Anal Mach Intell 38(3):431–446

    Article  Google Scholar 

  19. Qu L, Tian J, He S et al (2017) Deshadownet: A multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4067–4075

  20. Chen Z, Long C, Zhang L et al (2021) Canet: A context-aware network for shadow removal. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4743–4752

  21. Cun X, Pun CM, Shi C (2020) Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan. In: Proceedings of the AAAI conference on artificial intelligence, pp 10,680–10,687

  22. Hu X, Fu CW, Zhu L et al (2019) Direction-aware spatial context features for shadow detection and removal. IEEE Trans Pattern Anal Mach Intell 42(11):2795–2808

    Article  Google Scholar 

  23. Zhu JY, Park T, Isola P et al (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  24. Liu Z, Yin H, Mi Y et al (2021) Shadow removal by a lightness-guided network with training on unpaired data. IEEE Trans Image Process 30:1853–1865

    Article  Google Scholar 

  25. Liu Z, Yin H, Wu X et al (2021b) From shadow generation to shadow removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4927–4936

  26. Inoue N, Yamasaki T (2020) Learning from synthetic shadows for shadow detection and removal. IEEE Trans Circ Syst Video Technol 31(11):4187–4197

    Article  Google Scholar 

  27. Le H, Samaras D (2021) Physics-based shadow image decomposition for shadow removal. IEEE Trans Pattern Anal Mach Intell 44(12):9088–9101

    Article  Google Scholar 

  28. Fu L, Zhou C, Guo Q et al (2021) Auto-exposure fusion for single-image shadow removal. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10,571–10,580

  29. Zhang H, Wu C, Zhang Z et al (2022) Resnest: Split-attention networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2736–2746

  30. Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, et al (2018) Unet++: A nested u-net architecture for medical image segmentation. In: Deep Learning in medical image analysis and multimodal learning for clinical decision support: 4th International workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, Springer, pp 3–11

  31. Zhang Y, Tian Y, Kong Y et al (2020) Residual dense network for image restoration. IEEE Trans Pattern Anal Mach Intell 43(7):2480–2495

    Article  Google Scholar 

  32. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pmlr, pp 448–456

  33. Hou Q, Zhou D, Feng J (2021) Coordinate attention for efficient mobile network design. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13,713–13,722

  34. Mao X, Liu Y, Shen W, et al (2021) Deep residual fourier transformation for single image deblurring. arXiv preprint arXiv:2111.11745

  35. Xu H, Ma J, Zhang XP (2020) Mef-gan: Multi-exposure image fusion via generative adversarial networks. IEEE Trans Image Process 29:7203–7216

    Article  MATH  Google Scholar 

  36. Wang X, Girshick R, Gupta A et al (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7794–7803

  37. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  38. Yang Y, Qi Y (2021) Image super-resolution via channel attention and spatial graph convolutional network. Pattern Recognit 112(107):798

    Google Scholar 

  39. Wu P, Wang Z, Zheng B et al (2023) Aggn: Attention-based glioma grading network with multi-scale feature extraction and multi-modal information fusion. Comput Biol Med 152(106):457

    Google Scholar 

  40. Yu J, Lin Z, Yang J et al (2019) Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4471–4480

  41. Zhang H, Goodfellow I, Metaxas D et al (2019) Self-attention generative adversarial networks. In: International conference on machine learning, PMLR, pp 7354–7363

  42. Miyato T, Kataoka T, Koyama M et al (2018) Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957

  43. Odena A, Buckman J, Olsson C et al (2018) Is generator conditioning causally related to gan performance? In: International conference on machine learning, PMLR, pp 3849–3858

  44. Misra D (2019) Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:1908.08681

  45. Lai WS, Huang JB, Ahuja N et al (2018) Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Trans Pattern Anal Mach Intell 41(11):2599–2613

    Article  Google Scholar 

  46. Isola P, Zhu JY, Zhou T et al (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  47. Karras T, Laine S, Aila T (2019) A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4401–4410

  48. Wang Z, Bovik AC, Sheikh HR et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4):600–612

    Article  Google Scholar 

  49. Paszke A, Gross S, Massa F et al (2019) Pytorch: An imperative style, high-performance deep learning library

  50. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  51. Qin X, Zhang Z, Huang C et al (2020) U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognit 106(107):404

    Google Scholar 

  52. Gong H, Cosker D (2016) Interactive removal and ground truth for difficult shadow scenes. JOSA A 33(9):1798–1811

  53. Jie L, Zhang H (2022) Mgrln-net: Mask-guided residual learning network for joint single-image shadow detection and removal. In: Proceedings of the asian conference on computer vision, pp 4411–4427

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Zhejiang Normal University, Jinhua, People’s Republic of China

    Zile Xu & Xin Chen

Authors
  1. Zile Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Chen.

Ethics declarations

Conflict of interests

The authors declare that there are not any conflicts of interest in connection with this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, Z., Chen, X. A two-stage progressive shadow removal network. Appl Intell 53, 25296–25309 (2023). https://doi.org/10.1007/s10489-023-04856-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04856-2

Keywords

Search

Navigation