Skip to main content
SpringerLink
Log in

Reconstructing the Noise Variance Manifold for Image Denoising

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12354))

Included in the following conference series:

Abstract

Deep Convolutional Neural Networks (CNNs) have been successfully used in many low-level vision problems like image denoising. Although the conditional image generation techniques have led to large improvements in this task, there has been little effort in providing conditional generative adversarial networks (cGANs) with an explicit way of understanding the image noise for object-independent denoising reliable for real-world applications. The task of leveraging structures in the target space is unstable due to the complexity of patterns in natural scenes, so the presence of unnatural artifacts or over-smoothed image areas cannot be avoided. To fill the gap, in this work we introduce the idea of a cGAN which explicitly leverages structure in the image noise variance space. By learning directly a low dimensional manifold of the image noise variance, the generator promotes the removal from the noisy image only that information which spans this manifold. This idea brings many advantages while it can be appended at the end of any denoiser to significantly improve its performance. Based on our experiments, our model substantially outperforms existing state-of-the-art architectures, resulting in denoised images with less over-smoothing and better detail.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdelhamed, A., Brubaker, M.A., Brown, M.S.: Noise flow: noise modeling with conditional normalizing flows. ArXiv (2019)

    Google Scholar 

  2. Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  3. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study, pp. 1122–1131 (2017)

    Google Scholar 

  4. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. (TSP) 54, 4311–4322 (2006)

    Article  Google Scholar 

  5. Anaya, J., Barbu, A.: Renoir - a dataset for real low-light noise image reduction. J. Vis. Commun. Image Represent. 51, 144–154 (2018)

    Article  Google Scholar 

  6. Anwar, S., Barnes, N.: Real image denoising with feature attention. In: IEEE International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  7. Anwar, S., Porikli, F., Huynh, C.P.: Category-specific object image denoising. IEEE Trans. Image Process. 26(11), 5506–5518 (2017)

    Article  MathSciNet  Google Scholar 

  8. Berthelot, D., Schumm, T., Metz, L.: Began: Boundary equilibrium generative adversarial networks. ArXiv abs/1703.10717 (2017)

    Google Scholar 

  9. Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  10. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 60–65 (2005)

    Google Scholar 

  11. Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with bm3d?, pp. 2392–2399 (2012)

    Google Scholar 

  12. Bychkovsky, V., Paris, S., Chan, E., Durand, F.: Learning photographic global tonal adjustment with a database of input/output image pairs. In: The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition (2011)

    Google Scholar 

  13. Chen, F., Zhang, L., Yu, H.: External patch prior guided internal clustering for image denoising. In: IEEE International Conference on Computer Vision (ICCV), pp. 603–611 (2015)

    Google Scholar 

  14. Chen, J., Chen, J., Chao, H., Yang, M.: Image blind denoising with generative adversarial network based noise modeling. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  15. Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39, 1 (2016)

    Google Scholar 

  16. Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  17. Chrysos, G.G., Kossaifi, J., Zafeiriou, S.: Robust conditional generative adversarial networks. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  18. Cogswell, M., Ahmed, F., Girshick, R., Zitnick, L., Batra, D.: Reducing overfitting in deep networks by decorrelating representations. In: International Conference on Learning Representations (ICLR) (2016)

    Google Scholar 

  19. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  20. Dong, W., Li, X., Zhang, L., Shi, G.: Sparsity-based image denoising via dictionary learning and structural clustering. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 674–697 (2011)

    Google Scholar 

  21. Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22, 1620–1630 (2013)

    Article  MathSciNet  Google Scholar 

  22. Gonzalez, R.: Digital Image Processing 2Nd Ed. Prentice-Hall Of India Pvt. Limited (2002). https://books.google.co.uk/books?id=iyJOPgAACAAJ

  23. Gu, S., Li, Y., Gool, L.V., Timofte, R.: Self-guided network for fast image denoising. In: IEEE International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  24. Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2862–2869 (2014)

    Google Scholar 

  25. Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  26. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  27. Healey, G., Kondepudy, R.: Radiometric CCD camera calibration and noise estimation. IEEE Trans. Pattern Anal. Mach. Intell. 16, 267–276 (1994)

    Article  Google Scholar 

  28. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)

    Google Scholar 

  29. Izadi, S., Mirikharaji, Z., Zhao, M., Hamarneh, G.: Whitenner - blind image denoising via noise whiteness priors. In: International Conference on Computer Vision workshop on Visual Recognition for Medical Images (ICCV VRMI) (2019)

    Google Scholar 

  30. Jain, V., Sebastian, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)

    Google Scholar 

  31. Jia, X., Liu, S., Feng, X., Zhang, L.: Focnet: a fractional optimal control network for image denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  32. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  33. Kostadin, K., Foi, A., Katkovnik, V., Egiazarian, K.: BM3D image denoising with shape-adaptive principal component analysis (2009)

    Google Scholar 

  34. Krull, A., Buchholz, T.O., Jug, F.: Noise2void - learning denoising from single noisy images. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  35. Lebrun, M., Buades, A., Morel, J.M.: A nonlocal bayesian image denoising algorithm. SIAM J. Imaging Sci. 6, 1665–1688 (2013)

    Article  MathSciNet  Google Scholar 

  36. Lebrun, M., Colom, M., Morel, J.M.: The noise clinic: a blind image denoising algorithm. Image Process. Line (IPOL) 5, 1–54 (2015)

    Article  Google Scholar 

  37. Lehtinen, J., et al.: Noise2Noise: Learning image restoration without clean data. In: International Conference on Machine Learning (ICML), pp. 2965–2974 (2018)

    Google Scholar 

  38. Liu, C., Szeliski, R., Kang, S.B., Zitnick, C.L., Freeman, W.T.: Automatic estimation and removal of noise from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 30, 299–314 (2008)

    Article  Google Scholar 

  39. Liu, D., Wen, B., Fan, Y., Loy, C., Huang, T.S.: Non-local recurrent network for image restoration. In: International Conference on Neural Information Processing Systems (NIPS), pp. 1680–1689 (2018)

    Google Scholar 

  40. Liu, P., Zhang, H., Zhang, K., Lin, L., Zuog, W.: Multi-level wavelet-CNN for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 773–782 (2018)

    Google Scholar 

  41. Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)

    Google Scholar 

  42. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 416–423 (2001)

    Google Scholar 

  43. Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst Denoising With Kernel Prediction Networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  44. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)

  45. Nam, S., Hwang, Y., Matsushita, Y., Kim, S.J.: A holistic approach to cross-channel image noise modeling and its application to image denoising. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1683–1691 (2016)

    Google Scholar 

  46. Plotz, T., Roth, S.: Benchmarking denoising algorithms with real photographs. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2750–2759 (2017)

    Google Scholar 

  47. Plötz, T., Roth, S.: Neural nearest neighbors networks. In: Advances in Neural Information Processing Systems (NeurIPS) (2018)

    Google Scholar 

  48. Rasmus, A., Berglund, M., Honkala, M., Valpola, H., Raiko, T.: Semi-supervised learning with ladder networks. In: International Conference on Neural Information Processing Systems (NIPS), pp. 3546–3554 (2015)

    Google Scholar 

  49. Roth, S., Black, M.J.: Fields of experts. Int. J. Comput. Vision 82(2), 205–229 (2009)

    Article  Google Scholar 

  50. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training gans. In: International Conference on Neural Information Processing Systems (NIPS), pp. 2234–2242 (2016)

    Google Scholar 

  51. Tai, Y., Yang, J., Liu, X., Xu, C.: Memnet: A persistent memory network for image restoration. In: IEEE International Conference on Computer Vision (ICCV), pp. 4549–4557 (2017)

    Google Scholar 

  52. Tappen, M.F., Liu, C., Adelson, E.H., Freeman, W.T.: Learning gaussian conditional random fields for low-level vision, pp. 1–8, July 2007

    Google Scholar 

  53. Tripathi, S., Lipton, Z.C., Nguyen, T.Q.: Correction by projection: denoising images with generative adversarial networks. ArXiv abs/1803.04477 (2018)

    Google Scholar 

  54. Weiss, Y., Freeman, W.: What makes a good model of natural images?, pp. 1–8, July 2007

    Google Scholar 

  55. Xu, J., Osher, S.: Iterative regularization and nonlinear inverse scale space applied to wavelet-based denoising. IEEE Trans. Image Process. 16, 534–544 (2007)

    Article  MathSciNet  Google Scholar 

  56. Xu, J., Zhang, L., Zhang, D.: A trilateral weighted sparse coding scheme for real-world image denoising. In: European Conference on Computer Vision (ECCV), pp. 21–38 (2018)

    Google Scholar 

  57. Xu, J., Zhang, L., Zuo, W., Zhang, D., Feng, X.: Patch group based nonlocal self-similarity prior learning for image denoising. In: IEEE International Conference on Computer Vision (ICCV), pp. 244–252 (2015)

    Google Scholar 

  58. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)

    Article  MathSciNet  Google Scholar 

  59. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2808–2817 (2017)

    Google Scholar 

  60. Zhang, K., Zuo, W., Zhang, L.: FFDnet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. (TIP) 27(9), 4608–4622 (2018)

    Article  MathSciNet  Google Scholar 

  61. Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: IEEE International Conference on Computer Vision (ICCV), pp. 479–486 (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Huawei Noah’s Ark, London, UK

    Ioannis Marras, Ioannis Alexiou & Gregory Slabaugh

  2. Imperial College London, London, UK

    Grigorios G. Chrysos & Stefanos Zafeiriou

Authors
  1. Ioannis Marras
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Grigorios G. Chrysos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Alexiou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gregory Slabaugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stefanos Zafeiriou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioannis Marras .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 68597 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Marras, I., Chrysos, G.G., Alexiou, I., Slabaugh, G., Zafeiriou, S. (2020). Reconstructing the Noise Variance Manifold for Image Denoising. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12354. Springer, Cham. https://doi.org/10.1007/978-3-030-58545-7_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58545-7_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58544-0

  • Online ISBN: 978-3-030-58545-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation