Bidirectional Feature Pyramid Network with Recurrent Attention Residual Modules for Shadow Detection

  • Lei Zhu
  • Zijun Deng
  • Xiaowei Hu
  • Chi-Wing FuEmail author
  • Xuemiao Xu
  • Jing Qin
  • Pheng-Ann Heng
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11210)


This paper presents a network to detect shadows by exploring and combining global context in deep layers and local context in shallow layers of a deep convolutional neural network (CNN). There are two technical contributions in our network design. First, we formulate the recurrent attention residual (RAR) module to combine the contexts in two adjacent CNN layers and learn an attention map to select a residual and then refine the context features. Second, we develop a bidirectional feature pyramid network (BFPN) to aggregate shadow contexts spanned across different CNN layers by deploying two series of RAR modules in the network to iteratively combine and refine context features: one series to refine context features from deep to shallow layers, and another series from shallow to deep layers. Hence, we can better suppress false detections and enhance shadow details at the same time. We evaluate our network on two common shadow detection benchmark datasets: SBU and UCF. Experimental results show that our network outperforms the best existing method with 34.88% reduction on SBU and 34.57% reduction on UCF for the balance error rate.



The work is supported by the National Basic Program of China, 973 Program (Project no. 2015CB351706), the Research Grants Council of the Hong Kong Special Administrative Region (Project no. CUHK 14225616), Shenzhen Science and Technology Program (No. JCYJ20160429190300857 and JCYJ20170413162617606), the CUHK strategic recruitment fund, the NSFC (Grant No. 61272293, 61300137, 61472145, 61233012) and NSFG (Grant No. S2013010014973), RGC Fund (Grant No. CUHK14200915), Science and Technology Planning Major Project of Guangdong Province (Grant No. 2015A070711001), and Open Project Program of Guangdong Key Lab of Popular High Performance Computers and Shenzhen Key Lab of Service Computing and Applications (Grant No. SZU-GDPHPCL2015). Xiaowei Hu is funded by the Hong Kong Ph.D. Fellowship.

Supplementary material

474211_1_En_8_MOESM1_ESM.pdf (16.9 mb)
Supplementary material 1 (pdf 17336 KB)


  1. 1.
    Finlayson, G.D., Hordley, S.D., Lu, C., Drew, M.S.: On the removal of shadows from images. IEEE Trans. Pattern Anal. Mach. Intell. 28(1), 59–68 (2006)CrossRefGoogle Scholar
  2. 2.
    Finlayson, G.D., Drew, M.S., Lu, C.: Entropy minimization for shadow removal. Int. J. Comput. Vis. 85(1), 35–57 (2009)CrossRefGoogle Scholar
  3. 3.
    Khan, S.H., Bennamoun, M., Sohel, F., Togneri, R.: Automatic feature learning for robust shadow detection. In: CVPR, pp. 1939–1946 (2014)Google Scholar
  4. 4.
    Vicente, T.F.Y., Hou, L., Yu, C.-P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 816–832. Springer, Cham (2016). Scholar
  5. 5.
    Nguyen, V., Vicente, T.F.Y., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: ICCV, pp. 4510–4518 (2017)Google Scholar
  6. 6.
    Hu, X., Zhu, L., Fu, C.W., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection. In: CVPR, pp. 7454–7462 (2018)Google Scholar
  7. 7.
    Hu, X., Fu, C.W., Zhu, L., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection and removal. arXiv preprint arXiv:1805.04635 (2018)
  8. 8.
    Zhu, J., Samuel, K.G., Masood, S.Z., Tappen, M.F.: Learning to recognize shadows in monochromatic natural images. In: CVPR, pp. 223–230 (2010)Google Scholar
  9. 9.
    Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR, pp. 2117–2125 (2017)Google Scholar
  10. 10.
    Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: estimating surface normals from shadows under unknown reflectance and lighting conditions. In: ICCV, pp. 1693–1700 (2009)Google Scholar
  11. 11.
    Karsch, K., Hedau, V., Forsyth, D., Hoiem, D.: Rendering synthetic objects into legacy photographs. ACM Trans. Graph. (SIGGRAPH Asia) 30(6), 157:1–157:12 (2011)Google Scholar
  12. 12.
    Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Estimating natural illumination from a single outdoor image. In: ICCV, pp. 183–190 (2009)Google Scholar
  13. 13.
    Junejo, I.N., Foroosh, H.: Estimating geo-temporal location of stationary cameras using shadow trajectories. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 318–331. Springer, Heidelberg (2008). Scholar
  14. 14.
    Ecins, A., Fermuller, C., Aloimonos, Y.: Shadow free segmentation in still images using local density measure. In: ICCP, pp. 1–8 (2014)Google Scholar
  15. 15.
    Cucchiara, R., Grana, C., Piccardi, M., Prati, A.: Detecting moving objects, ghosts, and shadows in video streams. IEEE Trans. Pattern Anal. Mach. Intell. 25(10), 1337–1342 (2003)CrossRefGoogle Scholar
  16. 16.
    Nadimi, S., Bhanu, B.: Physical models for moving shadow and object detection in video. IEEE Trans. Pattern Anal. Mach. Intell. 26(8), 1079–1087 (2004)CrossRefGoogle Scholar
  17. 17.
    Tian, J., Qi, X., Qu, L., Tang, Y.: New spectrum ratio properties and features for shadow detection. Pattern Recogn. 51, 85–96 (2016)CrossRefGoogle Scholar
  18. 18.
    Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Detecting ground shadows in outdoor consumer photographs. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 322–335. Springer, Heidelberg (2010). Scholar
  19. 19.
    Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: CVPR, pp. 2033–2040 (2011)Google Scholar
  20. 20.
    Huang, X., Hua, G., Tumblin, J., Williams, L.: What characterizes a shadow boundary under the sun and sky? In: ICCV, pp. 898–905 (2011)Google Scholar
  21. 21.
    Vicente, Y., Tomas, F., Hoai, M., Samaras, D.: Leave-one-out kernel optimization for shadow detection. In: ICCV, pp. 3388–3396 (2015)Google Scholar
  22. 22.
    Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: CVPR, pp. 3147–3155 (2017)Google Scholar
  23. 23.
    Wang, F., et al.: Residual attention network for image classification. In: CVPR, pp. 3156–3164 (2017)Google Scholar
  24. 24.
    Hosseinzadeh, S., Shakeri, M., Zhang, H.: Fast shadow detection from a single image using a patched convolutional neural network. arXiv preprint arXiv:1709.09283 (2017)
  25. 25.
    Bell, S., Zitnick, C.L., Bala, K., Girshick, R.: Inside-outside net: detecting objects in context with skip pooling and recurrent neural networks. In: CVPR, pp. 2874–2883 (2016)Google Scholar
  26. 26.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)Google Scholar
  27. 27.
    Deng, Z., et al.: R\(^{3}\)Net: recurrent residual refinement network for saliency detection. In: IJCAI, pp. 684–690 (2018)Google Scholar
  28. 28.
    Li, G., Xie, Y., Lin, L., Yu, Y.: Instance-level salient object segmentation. In: CVPR, pp. 247–256 (2017)Google Scholar
  29. 29.
    Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR, pp. 5987–5995 (2017)Google Scholar
  30. 30.
    Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV, pp. 1395–1403 (2015)Google Scholar
  31. 31.
    Liu, W., Rabinovich, A., Berg, A.C.: ParseNet: looking wider to see better. arXiv preprint arXiv:1506.04579 (2015)
  32. 32.
    Krähenbühl, P., Koltun, V.: Efficient inference in fully connected CRFs with Gaussian edge potentials. In: NIPS, pp. 109–117 (2011)Google Scholar
  33. 33.
    Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.: DeshadowNet: a multi-context embedding deep network for shadow removal. In: CVPR, pp. 4067–4075 (2017)Google Scholar
  34. 34.
    Wang, T., Borji, A., Zhang, L., Zhang, P., Lu, H.: A stagewise refinement model for detecting salient objects in images. In: ICCV, pp. 4019–4028 (2017)Google Scholar
  35. 35.
    Zhang, P., Wang, D., Lu, H., Wang, H., Ruan, X.: Amulet: aggregating multi-level convolutional features for salient object detection. In: ICCV, pp. 202–211 (2017)Google Scholar
  36. 36.
    Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR, pp. 2881–2890 (2017)Google Scholar
  37. 37.
    Hou, Q., Cheng, M.M., Hu, X.W., Borji, A., Tu, Z., Torr, P.: Deeply supervised salient object detection with short connections. In: CVPR, pp. 3203–3212 (2017)Google Scholar
  38. 38.
    Hu, X., Zhu, L., Qin, J., Fu, C.W., Heng, P.A.: Recurrently aggregating deep features for salient object detection. In: AAAI, pp. 6943–6950 (2018)Google Scholar
  39. 39.
    Luo, Z., Mishra, A., Achkar, A., Eichel, J., Li, S., Jodoin, P.M.: Non-local deep features for salient object detection. In: CVPR, pp. 6609–6617 (2017)Google Scholar
  40. 40.
    Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: ICCV, pp. 212–221 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Lei Zhu
    • 1
    • 2
    • 5
  • Zijun Deng
    • 3
  • Xiaowei Hu
    • 1
  • Chi-Wing Fu
    • 1
    • 5
    Email author
  • Xuemiao Xu
    • 4
  • Jing Qin
    • 2
  • Pheng-Ann Heng
    • 1
    • 5
  1. 1.The Chinese University of Hong KongHong KongChina
  2. 2.The Hong Kong Polytechnic UniversityHong KongChina
  3. 3.South China University of TechnologyGuangzhouChina
  4. 4.Guangdong Provincial Key Lab of Computational Intelligence and Cyberspace InformationSouth China University of TechnologyGuangzhouChina
  5. 5.Shenzhen Key Laboratory of Virtual Reality and Human Interaction TechnologyShenzhen Institutes of Advanced Technology, Chinese Academy of SciencesShenzhenChina

Personalised recommendations