Stochastic Frequency Masking to Improve Super-Resolution and Denoising Networks

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12361)


Super-resolution and denoising are ill-posed yet fundamental image restoration tasks. In blind settings, the degradation kernel or the noise level are unknown. This makes restoration even more challenging, notably for learning-based methods, as they tend to overfit to the degradation seen during training. We present an analysis, in the frequency domain, of degradation-kernel overfitting in super-resolution and introduce a conditional learning perspective that extends to both super-resolution and denoising. Building on our formulation, we propose a stochastic frequency masking of images used in training to regularize the networks and address the overfitting problem. Our technique improves state-of-the-art methods on blind super-resolution with different synthetic kernels, real super-resolution, blind Gaussian denoising, and real-image denoising.


Image restoration Super-resolution Denoising Kernel overfitting 

Supplementary material

504471_1_En_44_MOESM1_ESM.pdf (21.1 mb)
Supplementary material 1 (pdf 21594 KB)


  1. 1.
    Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: CVPR Workshops (2017)Google Scholar
  2. 2.
    Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)zbMATHCrossRefGoogle Scholar
  3. 3.
    Ahmed, N., Natarajan, T., Rao, K.R.: Discrete cosine transform. IEEE Trans. Comput. 100(1), 90–93 (1974)MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Anwar, S., Barnes, N.: Real image denoising with feature attention. In: ICCV (2019)Google Scholar
  5. 5.
    Anwar, S., Huynh, C.P., Porikli, F.: Image deblurring with a class-specific prior. IEEE Trans. Pattern Anal. Mach. Intell. 41(9), 2112–2130 (2018)CrossRefGoogle Scholar
  6. 6.
    Batson, J., Royer, L.: Noise2Self: blind denoising by self-supervision. In: ICML (2019)Google Scholar
  7. 7.
    Beckouche, S., Starck, J.L., Fadili, J.: Astronomical image denoising using dictionary learning. Astron. Astrophys. 556, A132 (2013)CrossRefGoogle Scholar
  8. 8.
    Benazza-Benyahia, A., Pesquet, J.C.: Building robust wavelet estimators for multicomponent images using Stein’s principle. IEEE Trans. Image Process. 14(11), 1814–1830 (2005)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Burton, G.J., Moorhead, I.R.: Color and spatial structure in natural scenes. Appl. Opt. 26(1), 157–170 (1987)CrossRefGoogle Scholar
  10. 10.
    Cai, J., Zeng, H., Yong, H., Cao, Z., Zhang, L.: Toward real-world single image super-resolution: a new benchmark and a new model. In: ICCV (2019)Google Scholar
  11. 11.
    Chan, T.M., Zhang, J., Pu, J., Huang, H.: Neighbor embedding based super-resolution algorithm through edge detection and feature selection. Pattern Recogn. Lett. 30(5), 494–502 (2009)CrossRefGoogle Scholar
  12. 12.
    Chatterjee, P., Joshi, N., Kang, S.B., Matsushita, Y.: Noise suppression in low-light images through joint denoising and demosaicing. In: CVPR (2011)Google Scholar
  13. 13.
    Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2016)CrossRefGoogle Scholar
  14. 14.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22(4), 1620–1630 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Efrat, N., Glasner, D., Apartsin, A., Nadler, B., Levin, A.: Accurate blur models vs. image priors in single image super-resolution. In: ICCV (2013)Google Scholar
  17. 17.
    El Helou, M., Dümbgen, F., Süsstrunk, S.: AAM: an assessment metric of axial chromatic aberration. In: ICIP (2018)Google Scholar
  18. 18.
    El Helou, M., Süsstrunk, S.: Blind universal Bayesian image denoising with Gaussian noise level learning. IEEE Trans. Image Process. 29, 4885–4897 (2020)CrossRefGoogle Scholar
  19. 19.
    Field, D.J.: Relations between the statistics of natural images and the response properties of cortical cells. JOSA 4(12), 2379–2394 (1987)CrossRefGoogle Scholar
  20. 20.
    Foi, A., Trimeche, M., Katkovnik, V., Egiazarian, K.: Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process. 17(10), 1737–1754 (2008)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. Comput. Graph. Appl. 22(2), 56–65 (2002)CrossRefGoogle Scholar
  22. 22.
    Gu, J., Lu, H., Zuo, W.Z., Dong, C.: Blind super-resolution with iterative kernel correction. In: CVPR (2019)Google Scholar
  23. 23.
    Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: CVPR (2014)Google Scholar
  24. 24.
    Gunturk, B.K., Batur, A.U., Altunbasak, Y., Hayes, M.H., Mersereau, R.M.: Eigenface-domain super-resolution for face recognition. IEEE Trans. Image Process. 12(5), 597–606 (2003)CrossRefGoogle Scholar
  25. 25.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV (2016)Google Scholar
  26. 26.
    Khan, S.H., Hayat, M., Porikli, F.: Regularization of deep neural networks with spectral dropout. Neural Netw. 110, 82–90 (2019)CrossRefGoogle Scholar
  27. 27.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR (2016)Google Scholar
  28. 28.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  29. 29.
    Krull, A., Buchholz, T.O., Jug, F.: Noise2Void-learning denoising from single noisy images. In: CVPR (2019)Google Scholar
  30. 30.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR (2017)Google Scholar
  31. 31.
    Lefkimmiatis, S.: Universal denoising networks: a novel CNN architecture for image denoising. In: CVPR (2018)Google Scholar
  32. 32.
    Lehtinen, J., et al.: Noise2Noise: learning image restoration without clean data. In: ICML (2018)Google Scholar
  33. 33.
    Li, S., Yin, H., Fang, L.: Group-sparse representation with dictionary learning for medical image denoising and fusion. IEEE Trans. Biomed. Eng. 59(12), 3450–3459 (2012)CrossRefGoogle Scholar
  34. 34.
    Luisier, F., Blu, T., Unser, M.: Image denoising in mixed Poisson-Gaussian noise. IEEE Trans. Image Process. 20(3), 696–708 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Makitalo, M., Foi, A.: Optimal inversion of the generalized Anscombe transformation for Poisson-Gaussian noise. IEEE Trans. Image Process. 22(1), 91–103 (2012)MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Martin, D., Fowlkes, C., Tal, D., Malik, J., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV (2001)Google Scholar
  37. 37.
    Park, D.S., et al.: SpecAugment: a simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 (2019)
  38. 38.
    Peled, S., Yeshurun, Y.: Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging. Magn. Reson. Med. Official J. Int. Soc. Magn. Reson. Med. 45(1), 29–35 (2001)CrossRefGoogle Scholar
  39. 39.
    Plötz, T., Roth, S.: Neural nearest neighbors networks. In: NeurIPS (2018)Google Scholar
  40. 40.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234–241 (2015)Google Scholar
  41. 41.
    Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vision 82(2), 205 (2009)CrossRefGoogle Scholar
  42. 42.
    Schmidt, U., Roth, S.: Shrinkage fields for effective image restoration. In: CVPR (2014)Google Scholar
  43. 43.
    Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: CVPR (2015)Google Scholar
  44. 44.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: CVPR (2016)Google Scholar
  45. 45.
    Shi, W., et al.: Cardiac image super-resolution with global correspondence using multi-atlas patchMatch. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 9–16 (2013)Google Scholar
  46. 46.
    Shocher, A., Cohen, N., Irani, M.: Zero-shot super-resolution using deep internal learning. In: CVPR (2018)Google Scholar
  47. 47.
    Strang, G.: The discrete cosine transform. SIAM 41(1), 135–147 (1999)MathSciNetzbMATHCrossRefGoogle Scholar
  48. 48.
    Sun, J., Xu, Z., Shum, H.Y.: Image super-resolution using gradient profile prior. In: CVPR (2008)Google Scholar
  49. 49.
    Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: ICCV (2017)Google Scholar
  50. 50.
    Thornton, M.W., Atkinson, P.M., Holland, D.: Sub-pixel mapping of rural land cover objects from fine spatial resolution satellite sensor imagery using super-resolution pixel-swapping. Int. J. Remote Sens. 27(3), 473–491 (2006)CrossRefGoogle Scholar
  51. 51.
    Tolhurst, D., Tadmor, Y., Chao, T.: Amplitude spectra of natural images. Ophthalmic Physiol. Opt. 12(2), 229–232 (1992)CrossRefGoogle Scholar
  52. 52.
    Torralba, A., Oliva, A.: Statistics of natural image categories. Netw. Comput. Neural Syst. 14(3), 391–412 (2003)CrossRefGoogle Scholar
  53. 53.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: CVPR (2018)Google Scholar
  54. 54.
    Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: ECCV Workshops (2018)Google Scholar
  55. 55.
    Yang, C.Y., Ma, C., Yang, M.H.: Single-image super-resolution: a benchmark. In: ECCV (2014)Google Scholar
  56. 56.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  57. 57.
    Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: CVPR (2017)Google Scholar
  58. 58.
    Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Zhang, K., Zuo, W., Zhang, L.: Learning a single convolutional super-resolution network for multiple degradations. In: CVPR (2018)Google Scholar
  60. 60.
    Zhang, K., Zuo, W., Zhang, L.: Deep plug-and-play super-resolution for arbitrary blur kernels. In: CVPR (2019)Google Scholar
  61. 61.
    Zhang, X., Chen, Q., Ng, R., Koltu, V.: Zoom to learn, learn to zoom. In: ICCV (2019)Google Scholar
  62. 62.
    Zhang, Y., et al.: A poisson-Gaussian denoising dataset with real fluorescence microscopy images. In: CVPR (2019)Google Scholar
  63. 63.
    Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: ECCV (2018)Google Scholar
  64. 64.
    Zhou, R., Lahoud, F., El Helou, M., Süsstrunk, S.: A comparative study on wavelets and residuals in deep super resolution. Electron. Imaging 13, 135-1–135-6 (2019) CrossRefGoogle Scholar
  65. 65.
    Zhou, R., Süsstrunk, S.: Kernel modeling super-resolution on real low-resolution images. In: ICCV (2019)Google Scholar
  66. 66.
    Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: ICCV (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Computer and Communicatison Sciences, EPFLLausanneSwitzerland

Personalised recommendations