Skip to main content
SpringerLink
Log in

Unified Transformer Network for Multi-Weather Image Restoration

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13802))

Included in the following conference series:

Abstract

Vision based applications routinely involve restoration as a preprocessing step, making it impossible to have separate architectures for different types of weather restoration. But, most of the existing methods focus on weather specific application. Further, the methods for multi-weather image restoration have high computational constraints. To overcome these limitations, we propose a compact transformer based network, with 4.5M parameters (1/10\(^{th}\) of the existing method) for unified (simultaneous) removal of rain, snow and hazy effect with single set of trained parameters. We propose two parallel streams to handle the degradations: First, original resolution transformer stream (ORTS) focuses mainly on extracting fine level features through original scales of the inputs. Second, multi-level feature aggregation stream (MFAS) learns different sizes of the weather degradations. Further, it also uses coarse outputs from the first stream and utilizes edge boosting skip connections (EBSC) for propagating crucial edge details essential for image restoration. Finally, we present a memory replay training approach for generalization of the proposed network on multi-weather degraded scenarios. Substantial experiments on synthetic as well as real-world images, along with extensive ablation studies, demonstrate that the proposed method performs competitively with the existing methods for multi-weather image restoration. The code is provided at https://github.com/AshutoshKulkarni4998/UMWTransformer.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Chen, D., et al.: Gated context aggregation network for image dehazing and deraining. In: 2019 IEEE Winter Conference On Applications Of Computer Vision (WACV), pp. 1375–1383. IEEE (2019)

    Google Scholar 

  2. Chen, H., et al.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299–12310 (2021)

    Google Scholar 

  3. Chen, W.-T., Fang, H.-Y., Ding, J.-J., Tsai, C.-C., Kuo, S.-Y.: JSTASR: joint size and transparency-aware snow removal algorithm based on modified partial convolution and veiling effect removal. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 754–770. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_45

    Chapter  Google Scholar 

  4. Chen, W.T., et al.: All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4196–4205 (2021)

    Google Scholar 

  5. Chen, W.T., Huang, Z.K., Tsai, C.C., Yang, H.H., Ding, J.J., Kuo, S.Y.: Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17653–17662 (2022)

    Google Scholar 

  6. Chen, Y., Bai, W., Huang, Q., Xiao, J.: Efficient motion symbol detection and multi-kernel learning for aer object recognition. IEEE Transactions on Cognitive and Developmental Systems (2021). https://doi.org/10.1109/TCDS.2021.3122131

  7. Chen, Z., Wang, Y., Yang, Y., Liu, D.: Psd: Principled synthetic-to-real dehazing guided by physical priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7180–7189 (2021)

    Google Scholar 

  8. Deng, S., et al.: Detail-recovery image deraining via context aggregation networks. In: Proceedings of the IEEE/CVF Conference On Computer Vision And Pattern Recognition, pp. 14560–14569 (2020)

    Google Scholar 

  9. Dong, H., et al.: Multi-scale boosted dehazing network with dense feature fusion. In: Proceedings of the IEEE/CVF Conference On Computer Vision And Pattern Recognition, pp. 2157–2167 (2020)

    Google Scholar 

  10. Dosovitskiy, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  11. Eigen, D., Krishnan, D., Fergus, R.: Restoring an image taken through a window covered with dirt or rain. In: Proceedings of the IEEE International Conference On Computer Vision, pp. 633–640 (2013)

    Google Scholar 

  12. Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self-supervised monocular depth estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3828–3838 (2019)

    Google Scholar 

  13. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2010)

    Google Scholar 

  14. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  15. Jia, T., Li, J., Zhuo, L., Li, G.: Effective meta-attention dehazing networks for vision-based outdoor industrial systems. IEEE Trans. Industr. Inf. 18(3), 1511–1520 (2022). https://doi.org/10.1109/TII.2021.3059020

    Article  Google Scholar 

  16. Jiang, K., et al.: Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference On Computer Vision And Pattern Recognition, pp. 8346–8355 (2020)

    Google Scholar 

  17. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  18. Li, B., et al.: Benchmarking single-image dehazing and beyond. IEEE Trans. Image Process. 28(1), 492–505 (2019). https://doi.org/10.1109/TIP.2018.2867951

    Article  MathSciNet  MATH  Google Scholar 

  19. Li, J., Li, Y., Zhuo, L., Kuang, L., Yu, T.: Usid-net: Unsupervised single image dehazing network via disentangled representations. IEEE Trans. Multimedia (2022). https://doi.org/10.1109/TMM.2022.3163554

  20. Li, M., Cao, X., Zhao, Q., Zhang, L., Gao, C., Meng, D.: Video rain/snow removal by transformed online multiscale convolutional sparse coding. arXiv preprint arXiv:1909.06148 (2019)

  21. Li, R., Cheong, L.F., Tan, R.T.: Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1633–1642 (2019)

    Google Scholar 

  22. Li, R., Tan, R.T., Cheong, L.F.: All in one bad weather removal using architectural search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3175–3185 (2020)

    Google Scholar 

  23. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 262–277. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_16

    Chapter  Google Scholar 

  24. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition. pp. 2736–2744 (2016)

    Google Scholar 

  25. Liu, R.W., Guo, Y., Lu, Y., Chui, K.T., Gupta, B.B.: Deep network-enabled haze visibility enhancement for visual iot-driven intelligent transportation systems. IEEE Trans. Industrial Inform. (2022). https://doi.org/10.1109/TII.2022.3170594

  26. Liu, Y.F., Jaw, D.W., Huang, S.C., Hwang, J.N.: Desnownet: Context-aware deep network for snow removal. IEEE Trans. Image Process. 27(6), 3064–3073 (2018). https://doi.org/10.1109/TIP.2018.2806202

    Article  MathSciNet  Google Scholar 

  27. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015)

    Google Scholar 

  28. Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12179–12188 (2021)

    Google Scholar 

  29. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  30. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: A better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019)

    Google Scholar 

  31. Salazar-Colores, S., Cabal-Yepez, E., Ramos-Arreguin, J.M., Botella, G., Ledesma-Carrillo, L.M., Ledesma, S.: A fast image dehazing algorithm using morphological reconstruction. IEEE Trans. Image Process. 28(5), 2357–2366 (2018)

    Article  MathSciNet  Google Scholar 

  32. Shin, J., Park, H., Paik, J.: Region-based dehazing via dual-supervised triple-convolutional network. IEEE Trans. Multimedia 24, 245–260 (2022). https://doi.org/10.1109/TMM.2021.3050053

    Article  Google Scholar 

  33. Su, Z., Zhang, Y., Shi, J., Zhang, X.P.: Recurrent network knowledge distillation for image rain removal. IEEE Transactions on Cognitive and Developmental Systems (2021). https://doi.org/10.1109/TCDS.2021.3131045

  34. Valanarasu, J.M.J., Yasarla, R., Patel, V.M.: Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2353–2363 (2022)

    Google Scholar 

  35. Wang, Y., Liu, S., Chen, C., Zeng, B.: A hierarchical approach for rain or snow removing in a single color image. IEEE Trans. Image Process. 26(8), 3936–3950 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  36. Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: A general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17683–17693 (June 2022)

    Google Scholar 

  37. Xu, J., Zhao, W., Liu, P., Tang, X.: An improved guidance image based method to remove rain and snow in a single image. Comput. Inform. Sci. 5(3), 49 (2012)

    Google Scholar 

  38. Yi, D., Fang, H., Hua, Y., Su, J., Quddus, M., Han, J.: Improving synthetic to realistic semantic segmentation with parallel generative ensembles for autonomous urban driving. IEEE Trans. Cognitive Developm. Syst. (2021). https://doi.org/10.1109/TCDS.2021.3117925

  39. Yu, S., et al.: Content-adaptive rain and snow removal algorithms for single image. In: International Symposium on Neural Networks. pp. 439–448. Springer (2014). https://doi.org/10.1007/978-3-319-12436-0_49

  40. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: Efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5728–5739 (June 2022)

    Google Scholar 

  41. Zamir, S.W., et al.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021)

    Google Scholar 

  42. Zhai, M., Chen, L., Tung, F., He, J., Nawhal, M., Mori, G.: Lifelong gan: Continual learning for conditional image generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2759–2768 (2019)

    Google Scholar 

  43. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition, pp. 695–704 (2018)

    Google Scholar 

  44. Zhang, J., Tao, D.: Famed-net: A fast and accurate multi-scale end-to-end dehazing network. IEEE Trans. Image Process. 29, 72–84 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhao, S., Zhang, L., Shen, Y., Zhou, Y.: Refinednet: A weakly supervised refinement framework for single image dehazing. IEEE Trans. Image Process. 30, 3391–3404 (2021)

    Article  Google Scholar 

  46. Zheng, X., Liao, Y., Guo, W., Fu, X., Ding, X.: Single-image-based rain and snow removal using multi-guided filter. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 258–265. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_33

    Chapter  Google Scholar 

Download references

Acknowledgement

This work was supported by the Science and Engineering Research Board (DST-SERB), India, under Grant ECR/2018/001538.

Author information

Authors and Affiliations

  1. CVPR Lab, Indian Institute of Technology Ropar, Punjab, India

    Ashutosh Kulkarni, Shruti S. Phutke & Subrahmanyam Murala

Authors
  1. Ashutosh Kulkarni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shruti S. Phutke
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Subrahmanyam Murala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ashutosh Kulkarni .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 10261 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kulkarni, A., Phutke, S.S., Murala, S. (2023). Unified Transformer Network for Multi-Weather Image Restoration. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13802. Springer, Cham. https://doi.org/10.1007/978-3-031-25063-7_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25063-7_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25062-0

  • Online ISBN: 978-3-031-25063-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation