Advertisement

Stacking Networks Dynamically for Image Restoration Based on the Plug-and-Play Framework

Conference paper
  • 469 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)

Abstract

Recently, stacked networks show powerful performance in Image Restoration, such as challenging motion deblurring problems. However, the number of stacking levels is a hyper-parameter fine-tuned manually, making the stacking levels static during training without theoretical explanations for optimal settings. To address this challenge, we leverage the iterative process of the traditional plug-and-play method to provide a dynamic stacked network for Image Restoration. Specifically, a new degradation model with a novel update scheme is designed to integrate the deep neural network as the prior within the plug-and-play model. Compared with static stacked networks, our models are stacked dynamically during training via iterations, guided by a solid mathematical explanation. Theoretical proof on the convergence of the dynamic stacking process is provided. Experiments on the noise dataset BSD68, Set12, and motion blur dataset GoPro demonstrate that our framework outperforms the state-of-the-art in terms of PSNR and SSIM score without extra training process.

Keywords

Low-level vision Image restoration Plug-and-play 

Notes

Acknowledgement

This work was supported in part by grants from the National Natural Science Foundation of China (NSFC, No. 61973007, 61633002).

Supplementary material

504454_1_En_27_MOESM1_ESM.zip (57.5 mb)
Supplementary material 1 (zip 58832 KB)

References

  1. 1.
    Afonso, M.V., Bioucas-Dias, J.M., Figueiredo, M.A.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19(9), 2345–2356 (2010)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Alain, G., Bengio, Y.: What regularized auto-encoders learn from the data-generating distribution. J. Mach. Learn. Res. 15(1), 3563–3593 (2014)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Boyd, S., et al.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach. Learn. 3(1), 1–122 (2011)Google Scholar
  4. 4.
    Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with BM3D? In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2392–2399. IEEE (2012)Google Scholar
  5. 5.
    Chan, S.H., Wang, X., Elgendy, O.A.: Plug-and-play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imaging 3(1), 84–98 (2016)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2016)CrossRefGoogle Scholar
  7. 7.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)Google Scholar
  8. 8.
    Danielyan, A., Katkovnik, V., Egiazarian, K.: Image deblurring by augmented Lagrangian with BM3D frame prior. In: Workshop on Information Theoretic Methods in Science and Engineering (WITMSE), Tampere, Finland, pp. 16–18 (2010)Google Scholar
  9. 9.
    Dong, W., Wang, P., Yin, W., Shi, G., Wu, F., Lu, X.: Denoising prior driven deep neural network for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 41(10), 2305–2318 (2018)CrossRefGoogle Scholar
  10. 10.
    Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. (TOG) 35(6), 191 (2016)CrossRefGoogle Scholar
  11. 11.
    Gu, S., Timofte, R., Van Gool, L.: Integrating local and non-local denoiser priors for image restoration. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2923–2928. IEEE (2018)Google Scholar
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)Google Scholar
  13. 13.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  14. 14.
    Lehtinen, J., et al.: Noise2noise: learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018)
  15. 15.
    Liu, D., Wen, B., Fan, Y., Loy, C.C., Huang, T.S.: Non-local recurrent network for image restoration. In: Advances in Neural Information Processing Systems, pp. 1673–1682 (2018)Google Scholar
  16. 16.
    Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 773–782 (2018)Google Scholar
  17. 17.
    Liu, R., Fan, X., Cheng, S., Wang, X., Luo, Z.: Proximal alternating direction network: a globally converged deep unrolling framework. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)Google Scholar
  18. 18.
    Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3883–3891 (2017)Google Scholar
  19. 19.
    Ono, S.: Primal-dual plug-and-play image restoration. IEEE Sig. Process. Lett. 24(8), 1108–1112 (2017)CrossRefGoogle Scholar
  20. 20.
    Plötz, T., Roth, S.: Neural nearest neighbors networks. In: Advances in Neural Information Processing Systems, pp. 1087–1098 (2018)Google Scholar
  21. 21.
    Reehorst, E.T., Schniter, P.: Regularization by denoising: clarifications and new interpretations. IEEE Trans. Comput. Imaging 5(1), 52–67 (2018)CrossRefGoogle Scholar
  22. 22.
    Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vis. 82(2), 205 (2009)CrossRefGoogle Scholar
  23. 23.
    Santhanam, V., Morariu, V.I., Davis, L.S.: Generalized deep image to image regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5609–5619 (2017)Google Scholar
  24. 24.
    Sellent, A., Rother, C., Roth, S.: Stereo video deblurring. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 558–575. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_35CrossRefGoogle Scholar
  25. 25.
    Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 769–777 (2015)Google Scholar
  26. 26.
    Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174–8182 (2018)Google Scholar
  27. 27.
    Venkatakrishnan, S.V., Bouman, C.A., Wohlberg, B.: Plug-and-play priors for model based reconstruction. In: 2013 IEEE Global Conference on Signal and Information Processing, pp. 945–948. IEEE (2013)Google Scholar
  28. 28.
    Wang, S., Wen, B., Wu, J., Tao, D., Wang, Z.: Segmentation-aware image denoising without knowing true segmentation. arXiv preprint arXiv:1905.08965 (2019)
  29. 29.
    Xu, L., Ren, J.S., Liu, C., Jia, J.: Deep convolutional neural network for image deconvolution. In: Advances in Neural Information Processing Systems, pp. 1790–1798 (2014)Google Scholar
  30. 30.
    Yue, Z., Yong, H., Zhao, Q., Zhang, L., Meng, D.: Variational denoising network: toward blind noise modeling and removal. arXiv preprint arXiv:1908.11314 (2019)
  31. 31.
    Zhang, H., Dai, Y., Li, H., Koniusz, P.: Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5978–5986 (2019)Google Scholar
  32. 32.
    Zhang, J., Pan, J., Ren, J., Song, Y., Bao, L., Lau, R.W., Yang, M.H.: Dynamic scene deblurring using spatially variant recurrent neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2521–2529 (2018)Google Scholar
  33. 33.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938 (2017)Google Scholar
  35. 35.
    Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: 2011 International Conference on Computer Vision, pp. 479–486. IEEE (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Peking UniversityBeijingChina
  2. 2.University of Science and Technology BeijingBeijingChina

Personalised recommendations