Advertisement

Gated Fusion Network for Degraded Image Super Resolution

  • Xinyi Zhang
  • Hang Dong
  • Zhe Hu
  • Wei-Sheng Lai
  • Fei WangEmail author
  • Ming-Hsuan Yang
Article

Abstract

Single image super resolution aims to enhance image quality with respect to spatial content, which is a fundamental task in computer vision. In this work, we address the task of single frame super resolution with the presence of image degradation, e.g., blur, haze, or rain streaks. Due to the limitations of frame capturing and formation processes, image degradation is inevitable, and the artifacts would be exacerbated by super resolution methods. To address this problem, we propose a dual-branch convolutional neural network to extract base features and recovered features separately. The base features contain local and global information of the input image. On the other hand, the recovered features focus on the degraded regions and are used to remove the degradation. Those features are then fused through a recursive gate module to obtain sharp features for super resolution. By decomposing the feature extraction step into two task-independent streams, the dual-branch model can facilitate the training process by avoiding learning the mixed degradation all-in-one and thus enhance the final high-resolution prediction results. We evaluate the proposed method in three degradation scenarios. Experiments on these scenarios demonstrate that the proposed method performs more efficiently and favorably against the state-of-the-art approaches on benchmark datasets.

Keywords

Super resolution Image restoration Deep learning 

Notes

Acknowledgements

X. Zhang, H. Dong, and F. Wang are supported in part by National Major Science and Technology Projects of China Grant under No. 2019ZX01008103, National Natural Science Foundation of China (61603291), Natural Science Basic Research Plan in Shaanxi Province of China (Program No. 2018JM6057), and the Fundamental Research Funds for the Central Universities. W.-S. Lai and M.-H. Yang are supported in part by NSF CAREER Grant #1149783 and Gifts from Verisk, Adobe and Google.

References

  1. Agustsson, E., & Timofte, R. (2017). Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPR workshops Google Scholar
  2. Bai, Y., Zhang, Y., Ding, M., & Ghanem, B. (2018). Finding tiny faces in the wild with generative adversarial network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  3. Bao, W., Zhang, X., Yan, S., & Gao, Z. (2017). Iterative convolutional neural network for noisy image super-resolution. In IEEE international conference on image processing.Google Scholar
  4. Bascle, B., Blake, A., & Zisserman, A. (1996). Motion deblurring and super-resolution from an image sequence. In European conference on computer vision.Google Scholar
  5. Berman, D., Avidan, S., et al. (2016). Non-local image dehazing. IEEE conference on computer vision and pattern recognition (pp. 1674–1682).Google Scholar
  6. Chen, D., He, M., Fan, Q., Liao, J., Zhang, L., Hou, D., Yuan, L., Hua, G. (2019). Gated context aggregation network for image dehazing and deraining. In IEEE winter conference on applications of computer vision (pp. 1375–1383). IEEE.Google Scholar
  7. Cho, S., & Lee, S. (2009). Fast motion deblurring. ACM Transactions on Graphics, 28(5), 145.CrossRefGoogle Scholar
  8. Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307.CrossRefGoogle Scholar
  9. Fan, Z., Wu, H., Fu, X., Huang, Y., & Ding, X. (2017). Residual-guide network for single image deraining. In ACM international conference on multimedia.Google Scholar
  10. Fattal, R. (2008). Single image dehazing. ACM Transactions on Graphics, 27(3), 72.CrossRefGoogle Scholar
  11. Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The kitti vision benchmark suite. In IEEE conference on computer vision and pattern recognition.Google Scholar
  12. Godard, C., Mac Aodha, O., Firman, M., & Brostow, G. J. (2019). Digging into self-supervised monocular depth prediction. In IEEE international conference on computer vision.Google Scholar
  13. Gong, D., Yang, J., Liu, L., Zhang, Y., Reid, I. D., Shen, C., Van Den Hengel, A., & Shi, Q. (2017). From motion blur to motion flow: A deep learning solution for removing heterogeneous motion blur. In IEEE conference on computer vision and pattern recognition.Google Scholar
  14. Haris, M., Shakhnarovich, G., & Ukita, N. (2018). Deep backprojection networks for super-resolution. In IEEE conference on computer vision and pattern recognition.Google Scholar
  15. He, K., Sun, J., & Tang, X. (2011). Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(12), 2341–2353.CrossRefGoogle Scholar
  16. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE conference on computer vision and pattern recognition.Google Scholar
  17. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.CrossRefGoogle Scholar
  18. Hradiš, M., Kotera, J., Zemcík, P., & Šroubek, F. (2015). Convolutional neural networks for direct text deblurring. In British machine vision conference.Google Scholar
  19. Huang, J. B., Singh, A., & Ahuja, N. (2015). Single image super-resolution from transformed self-exemplars. In IEEE conference on computer vision and pattern recognition.Google Scholar
  20. Hu, Z., Xu, L., & Yang M. H. (2014). Joint depth estimation and camera shake removal from single blurry image. In IEEE conference on computer vision and pattern recognition.Google Scholar
  21. Jiang, T. X., Huang, T. Z., Zhao, X. L., Deng, L. J., & Wang, Y. (2017). A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors. In IEEE conference on computer vision and pattern recognition.Google Scholar
  22. Johnson, J., Alahi, A., & Li, F. F. (2016). Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision.Google Scholar
  23. Kim, J., Lee, J. K., & Lee, K. M. (2016a). Accurate image super-resolution using very deep convolutional networks. In IEEE conference on computer vision and pattern recognition.Google Scholar
  24. Kim, J., Lee, J. K., & Lee, K. M. (2016b). Deeply-recursive convolutional network for image super-resolution. In IEEE conference on computer vision and pattern recognition.Google Scholar
  25. Kim, T. H., Ahn, B., & Lee, K. M. (2013). Dynamic scene deblurring. In IEEE international conference on computer vision.Google Scholar
  26. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In International conference on learning representations.Google Scholar
  27. Köhler, R., Hirsch, M., Mohler, B., Schölkopf, B., & Harmeling, S. (2012). Recording and playback of camera shake: Benchmarking blind deconvolution with a real-world database. In European conference on computer vision.Google Scholar
  28. Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., & Matas, J. (2018). DeblurGAN: Blind motion deblurring using conditional adversarial networks. In IEEE conference on computer vision and pattern recognition.Google Scholar
  29. Lai, W. S., Huang, J. B., Ahuja, N., & Yang, M. H. (2019). Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(11), 2599–2613.CrossRefGoogle Scholar
  30. Lai, W. S., Huang, J. B., Hu, Z., Ahuja, N., & Yang, M. H. (2016). A comparative study for single image blind deblurring. In IEEE conference on computer vision and pattern recognition.Google Scholar
  31. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A. P., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  32. Li, B., Peng, X., Wang, Z., Xu, J., & Feng, D. (2017). Aod-net: All-in-one dehazing network. In IEEE international conference on computer vision.Google Scholar
  33. Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., et al. (2018a). Reside: A benchmark for single image dehazing. IEEE Transactions on Image Processing, 28(1), 492–505.CrossRefGoogle Scholar
  34. Li, G., He, X., Zhang, W., Chang, H., Dong, L., & Lin, L. (2018b). Non-locally enhanced encoder-decoder network for single image de-raining. arXiv:1808.01491.
  35. Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., & Meng, D. (2018c). Video rain streak removal by multiscale convolutional sparse coding. In IEEE conference on computer vision and pattern recognition.Google Scholar
  36. Li, X., Wu, J., Lin, Z., Liu, H., & Zha, H. (2018d). Recurrent squeeze-and-excitation context aggregation net for single image deraining. In European conference on computer vision.Google Scholar
  37. Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017). Enhanced deep residual networks for single image super-resolution. In CVPR workshops.Google Scholar
  38. Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., & Ma, Y. (2013). Robust recovery of subspace structures by low-rank representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 171–184.CrossRefGoogle Scholar
  39. Liu, J., Yang, W., Yang, S., & Guo, Z. (2018). Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In IEEE conference on computer vision and pattern recognition.Google Scholar
  40. Mairal, J., Bach, F., Ponce, J., Sapiro, G. (2009). Online dictionary learning for sparse coding. In International conference on machine learning.Google Scholar
  41. Mei, K., Jiang, A., Li, J., & Wang, M. (2018). Progressive feature fusion network for realistic image dehazing. In Asian conference on computer vision.Google Scholar
  42. Nah, S., Kim, T. H., & Lee, K. M. (2017). Deep multi-scale convolutional neural network for dynamic scene deblurring. In IEEE conference on computer vision and pattern recognition.Google Scholar
  43. Nimisha, T. M., Singh, A. K., & Rajagopalan, A. N. (2017). Blur-invariant deep learning for blind-deblurring. In IEEE international conference on computer vision.Google Scholar
  44. Noroozi, M., Chandramouli, P., & Favaro, P. (2017). Motion deblurring in the wild. In German conference on pattern recognition.Google Scholar
  45. Pan, J., Sun, D., Pfister, H., & Yang, M. H. (2016). Blind image deblurring using dark channel prior. In IEEE conference on computer vision and pattern recognition.Google Scholar
  46. Paramanand, C., & Rajagopalan, A. N. (2013). Non-uniform motion deblurring for bilayer scenes. In IEEE conference on computer vision and pattern recognition.Google Scholar
  47. Park, H., & Lee, K. M. (2017). Joint estimation of camera pose, depth, deblurring, and super-resolution from a blurred image sequence. In IEEE international conference on computer vision.Google Scholar
  48. Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv:1804.02767.
  49. Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., & Yang, M. H. (2016). Single image dehazing via multi-scale convolutional neural networks. In European conference on computer vision.Google Scholar
  50. Ren, W., Ma, L., Zhang, J., Pan, J., Cao, X., Liu, W., & Yang, M. H. (2018). Gated fusion network for single image dehazing. In IEEE conference on computer vision and pattern recognition.Google Scholar
  51. Reynolds, D. A., Quatieri, T. F., & Dunn, R. B. (2000). Speaker verification using adapted gaussian mixture models. Digital Signal Processing, 10(1–3), 19–41.CrossRefGoogle Scholar
  52. Sajjadi, M. S., Schölkopf, B., & Hirsch, M. (2017). Enhancenet: Single image super-resolution through automated texture synthesis. In IEEE international conference on computer vision.Google Scholar
  53. Schmidt, U., Rother, C., Nowozin, S., Jancsary, J., & Roth, S. (2013). Discriminative non-blind deblurring. In IEEE conference on computer vision and pattern recognition.Google Scholar
  54. Schmidt, U., Schelten, K., & Roth, S. (2011). Bayesian deblurring with integrated noise estimation. In IEEE conference on computer vision and pattern recognition.Google Scholar
  55. Schulter, S., Leistner, C., & Bischof, H. (2015). Fast and accurate image upscaling with super-resolution forests. In IEEE conference on computer vision and pattern recognition.Google Scholar
  56. Shan, Q., Jia, J., & Agarwala, A. (2008). High-quality motion deblurring from a single image. ACM Transactions on Graphics, 27(3), 73.CrossRefGoogle Scholar
  57. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016). Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  58. Su, S., Delbracio, M., Wang, J., Sapiro, G., Heidrich, W., & Wang, O. (2017). Deep video deblurring for hand-held cameras. In IEEE conference on computer vision and pattern recognition.Google Scholar
  59. Tai, Y., Yang, J., & Liu, X. (2017). Image super-resolution via deep recursive residual network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  60. Tan, R. T. (2008). Visibility in bad weather from a single image. In IEEE conference on computer vision and pattern recognition.Google Scholar
  61. Tao, X., Gao, H., Wang, Y., Shen, X., Wang, J., & Jia, J. (2018). Scale-recurrent network for deep image deblurring. In IEEE conference on computer vision and pattern recognition.Google Scholar
  62. Timofte, R., De Smet, V., & Van Gool, L. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution. In Asian conference on computer vision.Google Scholar
  63. Xu, L., Zheng, S., & Jia, J. (2013). Unnatural l0 sparse representation for natural image deblurring. In IEEE conference on computer vision and pattern recognition.Google Scholar
  64. Xu, X., Sun, D., Pan, J., Zhang, Y., Pfister, H., & Yang, M. H. (2017). Learning to super-resolve blurry face and text images. In IEEE international conference on computer vision.Google Scholar
  65. Yamaguchi, T., Fukuda, H., Furukawa, R., Kawasaki, H., & Sturm. P. (2010). Video deblurring and super-resolution technique for multiple moving objects. In Asian conference on computer vision.Google Scholar
  66. Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., & Yan, S. (2017). Deep joint rain detection and removal from a single image. In IEEE conference on computer vision and pattern recognition.Google Scholar
  67. Yu, X., Fernando, B., Hartley, R., & Porikli, F. (2018). Super-resolving very low-resolution face images with supplementary attributes. In IEEE conference on computer vision and pattern recognition.Google Scholar
  68. Zhang, H., & Patel, V. M. (2018a). Densely connected pyramid dehazing network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  69. Zhang, H., & Patel V. M. (2018b). Density-aware single image de-raining using a multi-stream dense network. In IEEE conference on computer vision and pattern recognition.Google Scholar
  70. Zhang, H., Yang, J., Zhang, Y., Nasrabadi, N. M., & Huang, T. S. (2011). Close the loop: Joint blind image restoration and recognition with sparse representation prior. In IEEE international conference on computer vision.Google Scholar
  71. Zhang, H., Sindagi, V., & Patel, V. M. (2017). Joint transmission map estimation and dehazing using deep networks. arXiv:1708.00581.
  72. Zhang, H., Sindagi, V., & Patel, V. M. (2019). Image de-raining using a conditional generative adversarial network. In Transactions on circuits and systems for video technology. IEEE.  https://doi.org/10.1109/TCSVT.2019.2920407.
  73. Zhang, L., Zhang, H., Shen, H., & Li, P. (2010). A super-resolution reconstruction algorithm for surveillance images. Signal Processing, 90(3), 848–859.CrossRefGoogle Scholar
  74. Zhang, X., Wang, F., Dong, H., & Guo, Y. (2018a). A deep encoder-decoder networks for joint deblurring and super-resolution. In IEEE international conference on acoustics, speech, and signal processing.Google Scholar
  75. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018b). Image super-resolution using very deep residual channel attention networks. In European conference on computer vision.Google Scholar
  76. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018c). Residual dense network for image super-resolution. In IEEE conference on computer vision and pattern recognition (pp. 2472–2481).Google Scholar
  77. Zou, W. W., & Yuen, P. C. (2012). Very low resolution face recognition problem. IEEE Transactions on Image Processing, 21(1), 327–340.MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2020

Authors and Affiliations

  1. 1.School of Software EngineeringXi’an Jiaotong UniversityXi’anChina
  2. 2.College of Artificial IntelligenceXi’an Jiaotong UniversityXi’anChina
  3. 3.Hikvision Research AmericaSanta ClaraUSA
  4. 4.Electrical Engineering and Computer ScienceUniversity of CaliforniaMercedUSA

Personalised recommendations