Advertisement

Reconstructing the Noise Variance Manifold for Image Denoising

Conference paper
  • 844 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12354)

Abstract

Deep Convolutional Neural Networks (CNNs) have been successfully used in many low-level vision problems like image denoising. Although the conditional image generation techniques have led to large improvements in this task, there has been little effort in providing conditional generative adversarial networks (cGANs) with an explicit way of understanding the image noise for object-independent denoising reliable for real-world applications. The task of leveraging structures in the target space is unstable due to the complexity of patterns in natural scenes, so the presence of unnatural artifacts or over-smoothed image areas cannot be avoided. To fill the gap, in this work we introduce the idea of a cGAN which explicitly leverages structure in the image noise variance space. By learning directly a low dimensional manifold of the image noise variance, the generator promotes the removal from the noisy image only that information which spans this manifold. This idea brings many advantages while it can be appended at the end of any denoiser to significantly improve its performance. Based on our experiments, our model substantially outperforms existing state-of-the-art architectures, resulting in denoised images with less over-smoothing and better detail.

Supplementary material

504446_1_En_36_MOESM1_ESM.pdf (67 mb)
Supplementary material 1 (pdf 68597 KB)

References

  1. 1.
    Abdelhamed, A., Brubaker, M.A., Brown, M.S.: Noise flow: noise modeling with conditional normalizing flows. ArXiv (2019)Google Scholar
  2. 2.
    Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  3. 3.
    Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study, pp. 1122–1131 (2017)Google Scholar
  4. 4.
    Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. (TSP) 54, 4311–4322 (2006)CrossRefGoogle Scholar
  5. 5.
    Anaya, J., Barbu, A.: Renoir - a dataset for real low-light noise image reduction. J. Vis. Commun. Image Represent. 51, 144–154 (2018)CrossRefGoogle Scholar
  6. 6.
    Anwar, S., Barnes, N.: Real image denoising with feature attention. In: IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  7. 7.
    Anwar, S., Porikli, F., Huynh, C.P.: Category-specific object image denoising. IEEE Trans. Image Process. 26(11), 5506–5518 (2017)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Berthelot, D., Schumm, T., Metz, L.: Began: Boundary equilibrium generative adversarial networks. ArXiv abs/1703.10717 (2017)Google Scholar
  9. 9.
    Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  10. 10.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 60–65 (2005)Google Scholar
  11. 11.
    Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with bm3d?, pp. 2392–2399 (2012)Google Scholar
  12. 12.
    Bychkovsky, V., Paris, S., Chan, E., Durand, F.: Learning photographic global tonal adjustment with a database of input/output image pairs. In: The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition (2011)Google Scholar
  13. 13.
    Chen, F., Zhang, L., Yu, H.: External patch prior guided internal clustering for image denoising. In: IEEE International Conference on Computer Vision (ICCV), pp. 603–611 (2015)Google Scholar
  14. 14.
    Chen, J., Chen, J., Chao, H., Yang, M.: Image blind denoising with generative adversarial network based noise modeling. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  15. 15.
    Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39, 1 (2016)Google Scholar
  16. 16.
    Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  17. 17.
    Chrysos, G.G., Kossaifi, J., Zafeiriou, S.: Robust conditional generative adversarial networks. In: International Conference on Learning Representations (ICLR) (2019)Google Scholar
  18. 18.
    Cogswell, M., Ahmed, F., Girshick, R., Zitnick, L., Batra, D.: Reducing overfitting in deep networks by decorrelating representations. In: International Conference on Learning Representations (ICLR) (2016)Google Scholar
  19. 19.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Dong, W., Li, X., Zhang, L., Shi, G.: Sparsity-based image denoising via dictionary learning and structural clustering. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 674–697 (2011)Google Scholar
  21. 21.
    Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22, 1620–1630 (2013)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Gonzalez, R.: Digital Image Processing 2Nd Ed. Prentice-Hall Of India Pvt. Limited (2002). https://books.google.co.uk/books?id=iyJOPgAACAAJ
  23. 23.
    Gu, S., Li, Y., Gool, L.V., Timofte, R.: Self-guided network for fast image denoising. In: IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  24. 24.
    Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2862–2869 (2014)Google Scholar
  25. 25.
    Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  26. 26.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)Google Scholar
  27. 27.
    Healey, G., Kondepudy, R.: Radiometric CCD camera calibration and noise estimation. IEEE Trans. Pattern Anal. Mach. Intell. 16, 267–276 (1994)CrossRefGoogle Scholar
  28. 28.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)Google Scholar
  29. 29.
    Izadi, S., Mirikharaji, Z., Zhao, M., Hamarneh, G.: Whitenner - blind image denoising via noise whiteness priors. In: International Conference on Computer Vision workshop on Visual Recognition for Medical Images (ICCV VRMI) (2019)Google Scholar
  30. 30.
    Jain, V., Sebastian, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)Google Scholar
  31. 31.
    Jia, X., Liu, S., Feng, X., Zhang, L.: Focnet: a fractional optimal control network for image denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  32. 32.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  33. 33.
    Kostadin, K., Foi, A., Katkovnik, V., Egiazarian, K.: BM3D image denoising with shape-adaptive principal component analysis (2009)Google Scholar
  34. 34.
    Krull, A., Buchholz, T.O., Jug, F.: Noise2void - learning denoising from single noisy images. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  35. 35.
    Lebrun, M., Buades, A., Morel, J.M.: A nonlocal bayesian image denoising algorithm. SIAM J. Imaging Sci. 6, 1665–1688 (2013)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Lebrun, M., Colom, M., Morel, J.M.: The noise clinic: a blind image denoising algorithm. Image Process. Line (IPOL) 5, 1–54 (2015)CrossRefGoogle Scholar
  37. 37.
    Lehtinen, J., et al.: Noise2Noise: Learning image restoration without clean data. In: International Conference on Machine Learning (ICML), pp. 2965–2974 (2018)Google Scholar
  38. 38.
    Liu, C., Szeliski, R., Kang, S.B., Zitnick, C.L., Freeman, W.T.: Automatic estimation and removal of noise from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 30, 299–314 (2008)CrossRefGoogle Scholar
  39. 39.
    Liu, D., Wen, B., Fan, Y., Loy, C., Huang, T.S.: Non-local recurrent network for image restoration. In: International Conference on Neural Information Processing Systems (NIPS), pp. 1680–1689 (2018)Google Scholar
  40. 40.
    Liu, P., Zhang, H., Zhang, K., Lin, L., Zuog, W.: Multi-level wavelet-CNN for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 773–782 (2018)Google Scholar
  41. 41.
    Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)Google Scholar
  42. 42.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 416–423 (2001)Google Scholar
  43. 43.
    Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst Denoising With Kernel Prediction Networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  44. 44.
    Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)
  45. 45.
    Nam, S., Hwang, Y., Matsushita, Y., Kim, S.J.: A holistic approach to cross-channel image noise modeling and its application to image denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1683–1691 (2016)Google Scholar
  46. 46.
    Plotz, T., Roth, S.: Benchmarking denoising algorithms with real photographs. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2750–2759 (2017)Google Scholar
  47. 47.
    Plötz, T., Roth, S.: Neural nearest neighbors networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2018)Google Scholar
  48. 48.
    Rasmus, A., Berglund, M., Honkala, M., Valpola, H., Raiko, T.: Semi-supervised learning with ladder networks. In: International Conference on Neural Information Processing Systems (NIPS), pp. 3546–3554 (2015)Google Scholar
  49. 49.
    Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vision 82(2), 205–229 (2009)CrossRefGoogle Scholar
  50. 50.
    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. In: International Conference on Neural Information Processing Systems (NIPS), pp. 2234–2242 (2016)Google Scholar
  51. 51.
    Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: IEEE International Conference on Computer Vision (ICCV), pp. 4549–4557 (2017)Google Scholar
  52. 52.
    Tappen, M.F., Liu, C., Adelson, E.H., Freeman, W.T.: Learning gaussian conditional random fields for low-level vision, pp. 1–8, July 2007Google Scholar
  53. 53.
    Tripathi, S., Lipton, Z.C., Nguyen, T.Q.: Correction by projection: denoising images with generative adversarial networks. ArXiv abs/1803.04477 (2018)Google Scholar
  54. 54.
    Weiss, Y., Freeman, W.: What makes a good model of natural images?, pp. 1–8, July 2007Google Scholar
  55. 55.
    Xu, J., Osher, S.: Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising. IEEE Trans. Image Process. 16, 534–544 (2007)MathSciNetCrossRefGoogle Scholar
  56. 56.
    Xu, J., Zhang, L., Zhang, D.: A trilateral weighted sparse coding scheme for real-world image denoising. In: European Conference on Computer Vision (ECCV), pp. 21–38 (2018)Google Scholar
  57. 57.
    Xu, J., Zhang, L., Zuo, W., Zhang, D., Feng, X.: Patch group based nonlocal self-similarity prior learning for image denoising. In: IEEE International Conference on Computer Vision (ICCV), pp. 244–252 (2015)Google Scholar
  58. 58.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2808–2817 (2017)Google Scholar
  60. 60.
    Zhang, K., Zuo, W., Zhang, L.: FFDnet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. (TIP) 27(9), 4608–4622 (2018)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: IEEE International Conference on Computer Vision (ICCV), pp. 479–486 (2011)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Huawei Noah’s ArkLondonUK
  2. 2.Imperial College LondonLondonUK

Personalised recommendations