Advertisement

Burst Denoising via Temporally Shifted Wavelet Transforms

Conference paper
  • 371 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)

Abstract

Mobile photography has made great strides in recent years. However, low light imaging remains a challenge. Long exposures can improve signal-to-noise ratio (SNR) but undesirable motion blur can occur when capturing dynamic scenes. Consequently, imaging pipelines often rely on computational photography to improve SNR by fusing multiple short exposures. Recent deep network-based methods have been shown to generate visually pleasing results by fusing these exposures in a sophisticated manner, but often at a higher computational cost.

We propose an end-to-end trainable burst denoising pipeline which jointly captures high-resolution and high-frequency deep features derived from wavelet transforms. In our model, precious local details are preserved in high-frequency sub-band features to enhance the final perceptual quality, while the low-frequency sub-band features carry structural information for faithful reconstruction and final objective quality. The model is designed to accommodate variable-length burst captures via temporal feature shifting while incurring only marginal computational overhead, and further trained with a realistic noise model for the generalization to real environments. Using these techniques, our method attains state-of-the-art performance on perceptual quality, while being an order of magnitude faster.

Keywords

Burst denoising Wavelet transform Deep learning 

Supplementary material

504454_1_En_15_MOESM1_ESM.zip (67.5 mb)
Supplementary material 1 (zip 69133 KB)

References

  1. 1.
    A Deep Look into the iPhone’s new Deep Fusion Feature. https://tinyurl.com/deepfusion. Accessed 04 Nov 2019
  2. 2.
    Night Sight: Seeing in the Dark on Pixel Phones. https://tinyurl.com/googlenightsight. Accessed 04 Nov 2019
  3. 3.
    Buades, T., Lou, Y., Morel, J.M., Tang, Z.: A note on multi-image denoising. In 2009 International Workshop on Local and Non-Local Approximation in Image Processing, pp. 1–15. IEEE (2009)Google Scholar
  4. 4.
    Liu, C., Freeman, W.T.: A high-quality video denoising algorithm based on reliable motion estimation. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 706–719. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15558-1_51CrossRefGoogle Scholar
  5. 5.
    Maggioni, M., Boracchi, G., Foi, A., Egiazarian, K.: Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms. IEEE Trans. Image Process. (TIP) 21(9), 3952–3966 (2012)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., Sun, J.: Fast burst images denoising. ACM Trans. Graphics (TOG) 33(6), 232 (2014)Google Scholar
  7. 7.
    Godard, C., Matzen, K., Uyttendaele, M.: Deep burst denoising. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 560–577. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01267-0_33CrossRefGoogle Scholar
  8. 8.
    Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks. In: CVPR, pp. 2502–2510 (2018)Google Scholar
  9. 9.
    Aittala, M., Durand, F.: Burst image deblurring using permutation invariant convolutional neural networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11212, pp. 748–764. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01237-3_45CrossRefGoogle Scholar
  10. 10.
    Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graphics (TOG) 35(6), 192 (2016)CrossRefGoogle Scholar
  11. 11.
    Kokkinos, F., Lefkimmiatis, S.: Iterative residual CNNS for burst photography applications. In: CVPR, pp. 5929–5938 (2019)Google Scholar
  12. 12.
    Dai, J., et al.: Deformable convolutional networks. In: ICCV, pp. 764–773 (2017)Google Scholar
  13. 13.
    Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: CVPR, pp. 1712–1722 (2019)Google Scholar
  14. 14.
    Ke, T.W., Maire, M., Yu, S.X.: Multigrid neural architectures. In: CVPR, pp. 6665–6673 (2017)Google Scholar
  15. 15.
    Wang, J., et al.: Deep High-resolution Representation Learning for Visual Recognition. arXiv preprint arXiv:1908.07919 (2019)
  16. 16.
    Chen, Y., et al.: Drop an octave: reducing spatial redundancy in convolutional neural networks with octave convolution. arXiv preprint arXiv:1904.05049 (2019)
  17. 17.
    Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228–6237 (2018)Google Scholar
  18. 18.
    Weickert, J.: Anisotropic diffusion in image processing. Teubner, Stuttgart (1998)zbMATHGoogle Scholar
  19. 19.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV, pp. 839–846 (1998)Google Scholar
  20. 20.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Antonini, M., Barlaud, M., Mathieu, P., Daubechies, I.: Image coding using wavelet transform. IEEE Trans. Image Process. (TIP) 1(2), 205–220 (1992)CrossRefGoogle Scholar
  22. 22.
    Portilla, J., Strela, V., Wainwright, M.J., Simoncelli, E.P.: Image denoising using scale mixtures of Gaussians in the wavelet domain. Trans. Img. Proc. 12(11), 1338–1351 (2003)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: CVPR, vol. 2, pp. 60–65. IEEE (2005)Google Scholar
  24. 24.
    Elad, M., Aharon, M.: Image denoising via learned dictionaries and sparse representation. In: CVPR, vol. 1, pp. 895–900. IEEE (2006)Google Scholar
  25. 25.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. (TIP) 16, 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. (TIP) 26(7), 3142–3155 (2017)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: CVPR Workshop, pp. 773–782 (2018)Google Scholar
  28. 28.
    Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: CVPR, pp. 3929–3938 (2017)Google Scholar
  29. 29.
    Laine, S., Lehtinen, J., Aila, T.: High-quality self-supervised deep image denoising. arXiv preprint arXiv:1901.10277 (2019)
  30. 30.
    Batson, J., Royer, L.: Noise2Self: blind denoising by self-supervision. In: ICML, pp. 524–533 (2019)Google Scholar
  31. 31.
    Anwar, S., Barnes, N.: Real image denoising with feature attention. In: ICCV (2019)Google Scholar
  32. 32.
    Cha, S., Moon, T.: Fully convolutional pixel adaptive image denoiser. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4160–4169 (2019)Google Scholar
  33. 33.
    Gu, S., et al.: Self-guided network for fast image denoising. In: ICCV (2019)Google Scholar
  34. 34.
    Arias, P., Morel, J.M.: Video denoising via empirical Bayesian estimation of space-time patches. J. Math. Imaging Vis. 60(1), 70–93 (2018)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Xu, J., Huang, Y., Liu, L., Zhu, F., Hou, X., Shao, L.: Noisy-as-clean: learning unsupervised denoising from the corrupted image. arXiv preprint arXiv:1906.06878 (2019)
  36. 36.
    Wang, J.Z.: Wavelets and imaging informatics: a review of the literature. J. Biomed. Inform. 34(2), 129–141 (2001)CrossRefGoogle Scholar
  37. 37.
    Williams, T., Li, R.: Wavelet pooling for convolutional neural networks. In: ICLR (2018)Google Scholar
  38. 38.
    Yoo, J., Uh, Y., Chun, S., Kang, B., Ha, J.W.: Photorealistic style transfer via wavelet transforms. In: ICCV (2019)Google Scholar
  39. 39.
    Deng, X., Yang, R., Xu, M., Dragotti, P.L.: Wavelet domain style transfer for an effective perception-distortion tradeoff in single image super-resolution (2019)Google Scholar
  40. 40.
    Kendall, A., Martirosyan, H., Dasgupta, S., Henry, P.: End-to-end learning of geometry and context for deep stereo regression. In: ICCV (2017)Google Scholar
  41. 41.
    Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: NeurIPS, pp. 3391–3401 (2017)Google Scholar
  42. 42.
    Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: NeurIPS, pp. 5099–5108 (2017)Google Scholar
  43. 43.
    Lin, J., Gan, C., Han, S.: TSM: temporal shift module for efficient video understanding. In: ICCV (2019)Google Scholar
  44. 44.
    Jaroensri, R., Biscarrat, C., Aittala, M., Durand, F.: Generating training data for denoising real RGB images via camera pipeline simulation. arXiv preprint arXiv:1904.08825 (2019)
  45. 45.
    Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: CVPR Workshop (2017)Google Scholar
  46. 46.
    Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task-oriented flow. Int. J. Comput. Vis. (IJCV) 127(8), 1106–1125 (2019)CrossRefGoogle Scholar
  47. 47.
    Xu, X., Li, M., Sun, W.: Learning deformable kernels for image and video denoising. arXiv preprint arXiv:1904.06903 (2019)
  48. 48.
    Steiner, B., et al.: PyTorch: An imperative style, high-performance deep learning library. NeurIPS 32 (2019)Google Scholar
  49. 49.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  50. 50.
    Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)Google Scholar
  51. 51.
    Zhou, Y., et al.: When AWGN-based denoiser meets real noises. arXiv preprint arXiv:1904.03485 (2019)
  52. 52.
    Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: CVPR, pp. 3291–3300 (2018)Google Scholar
  53. 53.
    Chen, C., Chen, Q., Do, M., Koltun, V.: Seeing motion in the dark. In: ICCV (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.FacebookSeattleUSA
  2. 2.CUNYNew YorkUSA

Personalised recommendations