Advertisement

Practical Deep Raw Image Denoising on Mobile Devices

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12351)

Abstract

Deep learning-based image denoising approaches have been extensively studied in recent years, prevailing in many public benchmark datasets. However, the stat-of-the-art networks are computationally too expensive to be directly applied on mobile devices. In this work, we propose a light-weight, efficient neural network-based raw image denoiser that runs smoothly on mainstream mobile devices, and produces high quality denoising results. Our key insights are twofold: (1) by measuring and estimating sensor noise level, a smaller network trained on synthetic sensor-specific data can out-perform larger ones trained on general data; (2) the large noise level variation under different ISO settings can be removed by a novel k-Sigma Transform, allowing a small network to efficiently handle a wide range of noise levels. We conduct extensive experiments to demonstrate the efficiency and accuracy of our approach. Our proposed mobile-friendly denoising model runs at \(\sim \)70 ms per megapixel on Qualcomm Snapdragon 855 chipset, and it is the basis of the night shot feature of several flagship smartphones released in 2019.

Supplementary material

504443_1_En_1_MOESM1_ESM.pdf (19.8 mb)
Supplementary material 1 (pdf 20276 KB)

References

  1. 1.
    Abdelhamed, A., Lin, S., Brown, M.S.: A high-quality denoising dataset for smartphone cameras. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  2. 2.
    Aharon, M., Elad, M., Bruckstein, A., et al.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Sig. Process. 54(11), 4311 (2006)CrossRefGoogle Scholar
  3. 3.
    Anaya, J., Barbu, A.: RENOIR-a dataset for real low-light image noise reduction. J. Vis. Commun. Image Represent. 51, 144–154 (2018)CrossRefGoogle Scholar
  4. 4.
    Anscombe, F.J.: The transformation of Poisson, binomial and negative-binomial data. Biometrika 35(3/4), 246–254 (1948)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11036–11045 (2019)Google Scholar
  6. 6.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 60–65. IEEE (2005)Google Scholar
  7. 7.
    Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising: can plain neural networks compete with BM3D? In: CVPR (2012)Google Scholar
  8. 8.
    Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3291–3300 (2018)Google Scholar
  9. 9.
    Chen, J., Chen, J., Chao, H., Yang, M.: Image blind denoising with generative adversarial network based noise modeling. In: CVPR (2018)Google Scholar
  10. 10.
    Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2017)CrossRefGoogle Scholar
  11. 11.
    Chollet, F.: Xception: deep learning with depthwise separable convolutions, October 2016. http://arxiv.org/abs/1610.02357
  12. 12.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image restoration by sparse 3D transform-domain collaborative filtering. In: Image Processing: Algorithms and Systems VI, vol. 6812, p. 681207. International Society for Optics and Photonics (2008)Google Scholar
  13. 13.
    Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15(12), 3736–3745 (2006)MathSciNetCrossRefGoogle Scholar
  14. 14.
    European Machine Vision Association.: Standard for Characterization of Image Sensors and Cameras (2010).  https://doi.org/10.1063/1.1518010
  15. 15.
    Foi, A., Alenius, S., Katkovnik, V., Egiazarian, K.: Noise measurement for raw-data of digital imaging sensors by automatic segmentation of nonuniform targets. IEEE Sens. J. 7(10), 1456–1461 (2007)CrossRefGoogle Scholar
  16. 16.
    Foi, A., Trimeche, M., Katkovnik, V., Egiazarian, K.: Practical Poissonian-Gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process. 17(10), 1737–1754 (2008)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Gharbi, M., Chaurasia, G., Paris, S., Durand, F.: Deep joint demosaicking and denoising. ACM Trans. Graph. (TOG) 35(6), 191 (2016)CrossRefGoogle Scholar
  18. 18.
    Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: CVPR (2014)Google Scholar
  19. 19.
    Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph. 35(6), 1–12 (2016).  https://doi.org/10.1145/2980179.2980254. http://dl.acm.org/citation.cfm?doid=2980179.2980254CrossRefGoogle Scholar
  20. 20.
    Hirakawa, K., Parks, T.W.: Joint demosaicing and denoising. IEEE Trans. Image Process. 15(8), 2146–2157 (2006)CrossRefGoogle Scholar
  21. 21.
    Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in neural information processing systems, pp. 769–776 (2009)Google Scholar
  22. 22.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  23. 23.
    Lehtinen, J., et al.: Noise2noise: learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018)
  24. 24.
    Liba, O., et al.: Handheld mobile photography in very low light. ACM Trans. Graph. 38(6) (2019).  https://doi.org/10.1145/3355089.3356508
  25. 25.
    Liu, C., Szeliski, R., Kang, S.B., Zitnick, C.L., Freeman, W.T.: Automatic estimation and removal of noise from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 299–314 (2008)CrossRefGoogle Scholar
  26. 26.
    Liu, J., et al.: Learning raw image denoising with Bayer pattern unification and Bayer preserving augmentation, April 2019. http://arxiv.org/abs/1904.12945
  27. 27.
    Liu, X., Tanaka, M., Okutomi, M.: Practical signal-dependent noise parameter estimation from a single noisy image. IEEE Trans. Image Process. 23(10), 4361–4371 (2014)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Mairal, J., Bach, F.R., Ponce, J., Sapiro, G., Zisserman, A.: Non-local sparse models for image restoration. In: ICCV, vol. 29, pp. 54–62. Citeseer (2009)Google Scholar
  29. 29.
    Makitalo, M., Foi, A.: Optimal inversion of the Anscombe transformation in low-count Poisson image denoising. IEEE Trans. Image Process. 20(1), 99–109 (2010)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: NeurIPS (2016)Google Scholar
  31. 31.
    Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks, December 2017. https://arxiv.org/abs/1712.02327
  32. 32.
    Portilla, J., Strela, V., Wainwright, M.J., Simoncelli, E.P.: Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Process. 12(11), 1338–1351 (2003)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  34. 34.
    Shi, G., Zifei, Y., Kai, Z., Wangmeng, Z., Lei, Z.: Toward convolutional blind denoising of real photographs. arXiv preprint arXiv:1807.04686 (2018)
  35. 35.
    Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472. IEEE (2017)Google Scholar
  36. 36.
    Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: Proceedings of the IEEE international Conference on Computer Vision, pp. 4539–4547 (2017)Google Scholar
  37. 37.
    Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2018)Google Scholar
  38. 38.
    Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, pp. 341–349 (2012)Google Scholar
  39. 39.
    Xu, J., Zhang, L., Zhang, D., Feng, X.: Multi-channel weighted nuclear norm minimization for real color image denoising. In: ICCV (2017)Google Scholar
  40. 40.
    Yair, N., Michaeli, T.: Multi-scale weighted nuclear norm image restoration. In: CVPR (2018)Google Scholar
  41. 41.
    Zhang, K., Zuo, W., Zhang, L.: FFDNet: toward a fast and flexible solution for CNN based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Zhou, Y., et al.: When AWGN-based denoiser meets real noises. arXiv preprint arXiv:1904.03485 (2019)
  43. 43.
    Zhou, Y., Liu, D., Huang, T.: Survey of face detection on low-quality images. In: 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), pp. 769–773. IEEE (2018)Google Scholar
  44. 44.
    Zhu, F., Chen, G., Heng, P.A.: From noise modeling to blind image denoising. In: CVPR (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Tsinghua UniversityBeijingChina
  2. 2.Megvii TechnologyBeijingChina

Personalised recommendations