Advertisement

From Shadow Segmentation to Shadow Removal

Conference paper
  • 585 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12356)

Abstract

The requirement for paired shadow and shadow-free images limits the size and diversity of shadow removal datasets and hinders the possibility of training large-scale, robust shadow removal algorithms. We propose a shadow removal method that can be trained using only shadow and non-shadow patches cropped from the shadow images themselves. Our method is trained via an adversarial framework, following a physical model of shadow formation. Our central contribution is a set of physics-based constraints that enables this adversarial training. Our method achieves competitive shadow removal results compared to state-of-the-art methods that are trained with fully paired shadow and shadow-free images. The advantages of our training regime are even more pronounced in shadow removal for videos. Our method can be fine-tuned on a testing video with only the shadow masks generated by a pre-trained shadow detector and outperforms state-of-the-art methods on this challenging test. We illustrate the advantages of our method on our proposed video shadow removal dataset.

Keywords

Shadow removal GAN Weakly-supervised Illumination model Unpaired Image-to-image 

Notes

Acknowledgements

This work was partially supported by the Partner University Fund, the SUNY2020 ITSC, and a gift from Adobe. Computational support provided by IACS and a GPU donation from NVIDIA. We thank Kumara Kahatapitiya and Cristina Mata for assistance with the manuscript.

Supplementary material

504452_1_En_16_MOESM1_ESM.pdf (17.2 mb)
Supplementary material 1 (pdf 17621 KB)

Supplementary material 2 (mp4 19388 KB)

Supplementary material 3 (mp4 19313 KB)

References

  1. 1.
    Arbel, E., Hel-Or, H.: Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1202–1216 (2011)CrossRefGoogle Scholar
  2. 2.
    Choi, Y., Choi, M.J., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2017)Google Scholar
  3. 3.
    Chuang, Y.Y., Goldman, D.B., Curless, B., Salesin, D.H., Szeliski, R.: Shadow matting and compositing. ACM Trans. Graph. 22(3), 494–500 (2003). Special Issue of SIGGRAPH 2003 ProceedingCrossRefGoogle Scholar
  4. 4.
    Dare, P.: Shadow analysis in high-resolution satellite imagery of urban areas. Photogram. Eng. Remote Sens. 71, 169–177 (2005).  https://doi.org/10.14358/PERS.71.2.169CrossRefGoogle Scholar
  5. 5.
    Ding, B., Long, C., Zhang, L., Xiao, C.: ARGAN: attentive recurrent generative adversarial network for shadow detection and removal. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10212–10221 (2019)Google Scholar
  6. 6.
    Drew, M.S.: Recovery of chromaticity image free from shadows via illumination invariance. In: IEEE Workshop on Color and Photometric Methods in Computer Vision, ICCV’03, pp. 32–39 (2003)Google Scholar
  7. 7.
    Finlayson, G., Drew, M.S.: 4-sensor camera calibration for image representation invariant to shading, shadows, lighting, and specularities. In: Proceedings of the International Conference on Computer Vision, vol. 2, pp. 473–480 (2001).  https://doi.org/10.1109/ICCV.2001.937663
  8. 8.
    Finlayson, G., Hordley, S., Lu, C., Drew, M.: On the removal of shadows from images. IEEE Trans. Pattern Anal. Mach. Intell. 28(1), 59–68 (2006)CrossRefGoogle Scholar
  9. 9.
    Finlayson, C., Hordley, S.D., Drew, M.S.: Removing shadows from images. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) Computer Vision – ECCV 2002. Lecture Notes in Computer Science, vol. 2353, pp. 823–836. Springer, Berlin, Heidelberg (2002).  https://doi.org/10.1007/3-540-47979-1_55CrossRefGoogle Scholar
  10. 10.
    Fredembach, C., Finlayson, G.D.: Hamiltonian path based shadow removal. In: BMVC (2005)Google Scholar
  11. 11.
    Gong, H., Cosker, D.: Interactive removal and ground truth for difficult shadow scenes. J. Opt. Soc. Am. A 33(9), 1798–1811 (2016).  https://doi.org/10.1364/JOSAA.33.001798. http://josaa.osa.org/abstract.cfm?URI=josaa-33-9-1798
  12. 12.
    Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)Google Scholar
  13. 13.
    Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2011)Google Scholar
  14. 14.
    Guo, R., Dai, Q., Hoiem, D.: Paired regions for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2956–2967 (2012)CrossRefGoogle Scholar
  15. 15.
    Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-ShadowGAN: learning to remove shadows from unpaired data. In: ICCV (2019)Google Scholar
  16. 16.
    Hu, X., Wang, T., Fu, C.W., Jiang, Y., Wang, Q., Heng, P.A.: Revisiting shadow detection: a new benchmark dataset for complex world. arXiv:abs/1911.06998 (2019)
  17. 17.
    Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  18. 18.
    KaewTrakulPong, P., Bowden, R.: An improved adaptive background mixture model for real- time tracking with shadow detection (2002)Google Scholar
  19. 19.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (2015)Google Scholar
  20. 20.
    Le, H., Goncalves, B., Samaras, D., Lynch, H.: Weakly labeling the antarctic: the penguin colony case. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)Google Scholar
  21. 21.
    Le, H., Nguyen, V., Yu, C.P., Samaras, D.: Geodesic distance histogram feature for video segmentation. In: ACCV (2016)Google Scholar
  22. 22.
    Le, H., Samaras, D.: Shadow removal via shadow image decomposition. In: Proceedings of the International Conference on Computer Vision (2019)Google Scholar
  23. 23.
    Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+D Net: training a shadow detector with adversarial shadow attenuation. In: Proceedings of the European Conference on Computer Vision (2018)Google Scholar
  24. 24.
    Le, H., Yu, C.P., Zelinsky, G., Samaras, D.: Co-localization with category-consistent features and geodesic distance propagation. In: ICCV 2017 Workshop on CEFRL: Compact and Efficient Feature Representation and Learning in Computer Vision (2017)Google Scholar
  25. 25.
    Li, Y., Tang, S., Zhang, R., Zhang, Y., Li, J., Yan, S.: Asymmetric gan for unpaired image-to-image translation. IEEE Trans. Image Process. 28, 5881–5896 (2019)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Liu, F., Gleicher, M.: Texture-consistent shadow removal. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) Computer Vision – ECCV 2008. Lecture Notes in Computer Science, vol. 5305, pp. 437–450. Springer, Berlin, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-88693-8_32CrossRefGoogle Scholar
  27. 27.
    Liu, H., Gu, X., Samaras, D.: Wasserstein GAN with quadratic transport cost. In: The IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  28. 28.
    Liu, H., Xianfeng, G., Samaras, D.: A two-step computation of the exact GAN Wasserstein distance. In: International Conference on Machine Learning, pp. 3165–3174 (2018)Google Scholar
  29. 29.
    Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. arXiv:abs/1703.00848 (2017)
  30. 30.
    Mescheder, L., Nowozin, S., Geiger, A.: Which training methods for GANs do actually converge? In: International Conference on Machine Learning (2018)Google Scholar
  31. 31.
    Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: International Conference on Machine Learning (2018)Google Scholar
  32. 32.
    Müller, T., Erdnüeß, B.: Brightness correction and shadow removal for video change detection with UAVs. In: Defense + Commercial Sensing (2019)Google Scholar
  33. 33.
    Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Estimating shadows with the bright channel cue. In: Kutulakos, K.N. (ed.) Trends and Topics in Computer Vision – ECCV 2010. Lecture Notes in Computer Science, vol. 6554, pp. 1–12. Springer, Berlin, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-35740-4_1CrossRefGoogle Scholar
  34. 34.
    Panagopoulos, A., Wang, C., Samaras, D., Paragios, N.: Simultaneous cast shadows, illumination and geometry inference using hypergraphs. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 437–449 (2013).  https://doi.org/10.1109/TPAMI.2012.110CrossRefGoogle Scholar
  35. 35.
    Porter, T., Duff, T.: Compositing digital images. Proc. ACM SIGGRAPH Conf. Comput. Graph. 18(3), 1–12 (1984)CrossRefGoogle Scholar
  36. 36.
    Prati, A., Mikic, I., Trivedi, M.M., Cucchiara, R.: Detecting moving shadows: algorithms and evaluation. IEEE Trans. Pattern Anal. Mach. Intell. 25, 918–923 (2003)CrossRefGoogle Scholar
  37. 37.
    Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.H.: DeshadowNet: a multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  38. 38.
    Shiting, W., Hong, Z.: Clustering-based shadow edge detection in a single color image. In: International Conference on Mechatronic Sciences, Electric Engineering and Computer, pp. 1038–1041 (2013).  https://doi.org/10.1109/MEC.2013.6885215
  39. 39.
    Shor, Y., Lischinski, D.: The shadow meets the mask: pyramid-based shadow removal. Comput. Graph. Forum 27(2), 577–586 (2008)CrossRefGoogle Scholar
  40. 40.
    Smith, A.R., Blinn, J.F.: Blue screen matting. In: Proceedings of the ACM SIGGRAPH Conference on Computer Graphics (1996)Google Scholar
  41. 41.
    Su, N., Zhang, Y., Tian, S., Yan, Y., Miao, X.: Shadow detection and removal for occluded object information recovery in urban high-resolution panchromatic satellite images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 9, 2568–2582 (2016)CrossRefGoogle Scholar
  42. 42.
    Thanh-Tung, H., Tran, T., Venkatesh, S.: Improving generalization and stability of generative adversarial networks. In: International Conference on Learning Representations (2019)Google Scholar
  43. 43.
    Vicente, T.F.Y., Hoai, M., Samaras, D.: Noisy label recovery for shadow detection in unfamiliar domains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  44. 44.
    Vicente, T.F.Y., Hoai, M., Samaras, D.: Leave-one-out kernel optimization for shadow detection and removal. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(3), 682–695 (2018)CrossRefGoogle Scholar
  45. 45.
    Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: Proceedings of the European Conference on Computer Vision (2016)Google Scholar
  46. 46.
    Vicente, T.F.Y., Samaras, D.: Single image shadow removal via neighbor-based region relighting. In: Proceedings of the European Conference on Computer Vision Workshops (2014)Google Scholar
  47. 47.
    Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  48. 48.
    Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: CVPR (2020)Google Scholar
  49. 49.
    Wright, S.: Digital Compositing for Film and Video. Focal Press (2001)Google Scholar
  50. 50.
    Wu, Q., Zhang, W.,Vijay Kumar, B.V.K.: Strong shadow removal via patch-based shadow edge detection. In: 2012 IEEE International Conference on Robotics and Automation, pp. 2177–2182 (2012)Google Scholar
  51. 51.
    Wu, T.P., Tang, C.K.: A Bayesian approach for shadow extraction from a single image. In: Tenth IEEE International Conference on Computer Vision (ICCV’05), vol. 1 1, pp. 480–487 (2005)Google Scholar
  52. 52.
    Wu, T.P., Tang, C.K., Brown, M.S., Shum, H.Y.: Natural shadow matting. ACM Trans. Graph. 26, 2 (2007).  https://doi.org/10.1145/1243980.1243982. http://doi.acm.org/10.1145/1243980.1243982
  53. 53.
    Yang, Q., Tan, K., Ahuja, N.: Shadow removal using bilateral filtering. IEEE Trans. Image Process. 21, 4361–4368 (2012)MathSciNetCrossRefGoogle Scholar
  54. 54.
    Yi, Z., Zhang, H., Tan, P., Gong, M.: DualGAN: unsupervised dual learning for image-to-image translation. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2868–2876 (2017)Google Scholar
  55. 55.
    Zhang, L., Long, C., Zhang, X., Xiao, C.: RIS-GAN: explore residual and illumination with generative adversarial networks for shadow removal. In: AAAI Conference on Artificial Intelligence (AAAI) (2020)Google Scholar
  56. 56.
    Zhang, W., Zhao, X., Morvan, J.M., Chen, L.: Improving shadow suppression for illumination robust face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41, 611–624 (2019)CrossRefGoogle Scholar
  57. 57.
    Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.H.: Distraction-aware shadow detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5162–5171 (2019)Google Scholar
  58. 58.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar
  59. 59.
    Zhu, L., et al.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of the European Conference on Computer Vision (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Stony Brook UniversityStony BrookUSA

Personalised recommendations