Abstract
Night imaging with modern smartphone cameras is troublesome due to low photon count and unavoidable noise in the imaging system. Directly adjusting exposure time and ISO ratings cannot obtain sharp and noise-free images at the same time in low-light conditions. Though many methods have been proposed to enhance noisy or blurry night images, their performances on real-world night photos are still unsatisfactory due to two main reasons: 1) Limited information in a single image and 2) Domain gap between synthetic training images and real-world photos (e.g., differences in blur area and resolution). To exploit the information from successive long- and short-exposure images, we propose a learning-based pipeline to fuse them. A D2HNet framework is developed to recover a high-quality image by deblurring and enhancing a long-exposure image under the guidance of a short-exposure image. To shrink the domain gap, we leverage a two-phase DeblurNet-EnhanceNet architecture, which performs accurate blur removal on a fixed low resolution so that it is able to handle large ranges of blur in different resolution inputs. In addition, we synthesize a D2-Dataset from HD videos and experiment on it. The results on the validation set and real photos demonstrate our methods achieve better visual quality and state-of-the-art quantitative scores. The D2HNet codes and D2-Dataset can be found at https://github.com/zhaoyuzhi/D2HNet.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We thank Chao Wang in the SenseTime Research for helping capture the image.
References
Abdelhamed, A., Brubaker, M.A., Brown, M.S.: Noise flow: noise modeling with conditional normalizing flows. In: Proceedings ICCV, pp. 3165–3173 (2019)
Anwar, S., Barnes, N.: Real image denoising with feature attention. In: Proceedings ICCV, pp. 3155–3164 (2019)
Brooks, T., Mildenhall, B., Xue, T., Chen, J., Sharlet, D., Barron, J.T.: Unprocessing images for learned raw denoising. In: Proceedings CVPR, pp. 11036–11045 (2019)
Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: Proceedings CVPR, vol. 2, pp. 60–65 (2005)
Byun, J., Cha, S., Moon, T.: FBI-denoiser: fast blind image denoiser for poisson-gaussian noise. In: Proceedings CVPR, pp. 5768–5777 (2021)
Chakrabarti, A.: A neural approach to blind motion deblurring. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 221–235. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_14
Chan, K.C., Wang, X., Yu, K., Dong, C., Loy, C.C.: BasicVSR: the search for essential components in video super-resolution and beyond. In: Proceedings CVPR, pp. 4947–4956 (2021)
Chan, K.C., Zhou, S., Xu, X., Loy, C.C.: BasicVSR++: improving video super-resolution with enhanced propagation and alignment. In: Proceedings CVPR, pp. 5972–5981 (2022)
Chang, M., Feng, H., Xu, Z., Li, Q.: Low-light image restoration with short-and long-exposure raw pairs. IEEE Trans. Multimedia 24, 702–714 (2021)
Chen, C., Chen, Q., Xu, J., Koltun, V.: Learning to see in the dark. In: Proceedings CVPR, pp. 3291–3300 (2018)
Chen, J., Chen, J., Chao, H., Yang, M.: Image blind denoising with generative adversarial network based noise modeling. In: Proceedings CVPR, pp. 3155–3164 (2018)
Chen, L., Lu, X., Zhang, J., Chu, X., Chen, C.: Hinet: half instance normalization network for image restoration. In: Proceedings CVPRW, pp. 182–192 (2021)
Chen, Y., Pock, T.: Trainable nonlinear reaction diffusion: a flexible framework for fast and effective image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1256–1272 (2016)
Cheng, S., Wang, Y., Huang, H., Liu, D., Fan, H., Liu, S.: NbNet: noise basis learning for image denoising with subspace projection. In: Proceedings CVPR, pp. 4896–4906 (2021)
Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: Proceedings ICCV, pp. 4641–4650 (2021)
Choi, B.D., Jung, S.W., Ko, S.J.: Motion-blur-free camera system splitting exposure time. IEEE Trans. Consum. Electron. 54(3), 981–986 (2008)
Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)
Dai, J., et al.: Deformable convolutional networks. In: Proceedings ICCV, pp. 764–773 (2017)
Deng, J., Wang, L., Pu, S., Zhuo, C.: Spatio-temporal deformable convolution for compressed video quality enhancement. In: Proceedings, AAAI. vol. 34, pp. 10696–10703 (2020)
Dudhane, A., Zamir, S.W., Khan, S., Khan, F.S., Yang, M.H.: Burst image restoration and enhancement. In: Proceedings CVPR, pp. 5759–5768 (2022)
Gao, H., Tao, X., Shen, X., Jia, J.: Dynamic scene deblurring with parameter selective sharing and nested skip connections. In: Proceedings CVPR, pp. 3848–3856 (2019)
Godard, C., Matzen, K., Uyttendaele, M.: Deep burst denoising. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 560–577. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01267-0_33
Gong, D., et al.: From motion blur to motion flow: a deep learning solution for removing heterogeneous motion blur. In: Proceedings CVPR, pp. 2319–2328 (2017)
Gu, C., Lu, X., He, Y., Zhang, C.: Blur removal via blurred-noisy image pair. IEEE Trans. Image Process. 30, 345–359 (2020)
Gu, S., Li, Y., Gool, L.V., Timofte, R.: Self-guided network for fast image denoising. In: Proceedings ICCV, pp. 2511–2520 (2019)
Guo, S., Yan, Z., Zhang, K., Zuo, W., Zhang, L.: Toward convolutional blind denoising of real photographs. In: Proceedings CVPR, pp. 1712–1722 (2019)
Guo, S., Yang, X., Ma, J., Ren, G., Zhang, L.: A differentiable two-stage alignment scheme for burst image reconstruction with large shift. In: Proceedings CVPR, pp. 17472–17481 (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings CVPR, pp. 770–778 (2016)
Hu, X., et al.: Pseudo 3D auto-correlation network for real image denoising. In: Proceedings CVPR, pp. 16175–16184 (2021)
Ji, S.W., et al.: XYDeblur: divide and conquer for single image deblurring. In: Proceedings CVPR, pp. 17421–17430 (2022)
Jiang, H., Sun, D., Jampani, V., Yang, M.H., Learned-Miller, E., Kautz, J.: Super slomo: high quality estimation of multiple intermediate frames for video interpolation. In: Proceedings CVPR, pp. 9000–9008 (2018)
Karadeniz, A.S., Erdem, E., Erdem, A.: Burst photography for learning to enhance extremely dark images. IEEE Trans. Image Process. 30, 9372–9385 (2021)
Kim, Y., Soh, J.W., Park, G.Y., Cho, N.I.: Transfer learning from synthetic to real-noise denoising with adaptive instance normalization. In: Proceedings CVPR, pp. 3482–3492 (2020)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings ICLR (2014)
Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-Laplacian priors. In: Proceedings NeurIPS, pp. 1033–1041 (2009)
Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: Deblurgan: blind motion deblurring using conditional adversarial networks. In: Proceedings CVPR, pp. 8183–8192 (2018)
Kupyn, O., Martyniuk, T., Wu, J., Wang, Z.: Deblurgan-v2: deblurring (orders-of-magnitude) faster and better. In: Proceedings ICCV, pp. 8878–8887 (2019)
Lamba, M., Mitra, K.: Restoring extremely dark images in real time. In: Proceedings CVPR, pp. 3487–3497 (2021)
Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Efficient marginal likelihood optimization in blind deconvolution. In: Proceedings CVPR, pp. 2657–2664 (2011)
Li, H., Wu, X.J.: DenseFuse: a fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2018)
Li, M., Liu, J., Yang, W., Sun, X., Guo, Z.: Structure-revealing low-light image enhancement via robust retinex model. IEEE Trans. Image Process. 27(6), 2828–2841 (2018)
Liba, O., et al.: Handheld mobile photography in very low light. ACM Trans. Graph. 38(6), 1–16 (2019)
Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: Proceedings CVPRW, pp. 773–782 (2018)
Liu, W., Yan, Q., Zhao, Y.: Densely self-guided wavelet network for image denoising. In: Proceedings CVPRW, pp. 432–433 (2020)
Liu, Y., et al.: Invertible denoising network: a light solution for real noise removal. In: Proceedings CVPR, pp. 13365–13374 (2021)
Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., Sun, J.: Fast burst images denoising. ACM Trans. Graph. 33(6), 1–9 (2014)
Luo, Z., et al.: EBSR: feature enhanced burst super-resolution with deformable alignment. In: Proceedings CVPRW, pp. 471–478 (2021)
Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Proceedings NeurIPS, pp. 2802–2810 (2016)
Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks. In: Proceedings CVPR, pp. 2502–2510 (2018)
Mustaniemi, J., Kannala, J., Matas, J., Särkkä, S., Heikkilä, J.: Lsd\(_2\) - joint denoising and deblurring of short and long exposure images with convolutional neural networks. In: Proceedings BMVC (2020)
Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: Proceedings CVPR, pp. 3883–3891 (2017)
Nimisha, T.M., Kumar Singh, A., Rajagopalan, A.N.: Blur-invariant deep learning for blind-deblurring. In: Proceedings ICCV, pp. 4752–4760 (2017)
Park, D., Kang, D.U., Kim, J., Chun, S.Y.: Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 327–343. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_20
Purohit, K., Rajagopalan, A.: Region-adaptive dense network for efficient motion deblurring. In: Proceedings AAAI, vol. 34, pp. 11882–11889 (2020)
Ren, C., He, X., Wang, C., Zhao, Z.: Adaptive consistency prior based deep network for image denoising. In: Proceedings CVPR, pp. 8596–8606 (2021)
Richardson, W.H.: Bayesian-based iterative method of image restoration. JoSA 62(1), 55–59 (1972)
Rim, J., Lee, H., Won, J., Cho, S.: Real-world blur dataset for learning and benchmarking deblurring algorithms. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 184–201. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_12
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Proceedings MICCAI, pp. 234–241 (2015)
Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992)
Simoncelli, E.P., Adelson, E.H.: Noise removal via Bayesian wavelet coring. In: Proceedings ICIP, vol. 1, pp. 379–382 (1996)
Son, C.H., Choo, H., Park, H.M.: Image-pair-based deblurring with spatially varying norms and noisy image updating. J. Vis. Comm. Image Rep. 24(8), 1303–1315 (2013)
Son, C.H., Park, H.M.: A pair of noisy/blurry patches-based PSF estimation and channel-dependent deblurring. IEEE Trans. Consum. Electron. 57(4), 1791–1799 (2011)
Suin, M., Purohit, K., Rajagopalan, A.: Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In: Proceedings, pp. 3606–3615 (2020)
Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: Proceedings, CVPR. pp. 769–777 (2015)
Tai, Y., Yang, J., Liu, X., Xu, C.: MemNet: a persistent memory network for image restoration. In: Proceedings ICCV, pp. 4539–4547 (2017)
Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: Proceedings CVPR, pp. 8174–8182 (2018)
Tian, Y., Zhang, Y., Fu, Y., Xu, C.: TDAN: temporally-deformable alignment network for video super-resolution. In: Proceedings CVPR, pp. 3360–3369 (2020)
Tico, M., Gelfand, N., Pulli, K.: Motion-blur-free exposure fusion. In: Proceedings ICIP, pp. 3321–3324 (2010)
Wang, X., Chan, K.C., Yu, K., Dong, C., Change Loy, C.: EDVR: video restoration with enhanced deformable convolutional networks. In: Proceedings CVPRW, pp. 1–10 (2019)
Wang, Y., et al.: Progressive retinex: mutually reinforced illumination-noise perception network for low-light image enhancement. In: Proceedings ACM MM, pp. 2015–2023 (2019)
Wang, Y., Huang, H., Xu, Q., Liu, J., Liu, Y., Wang, J.: Practical deep raw image denoising on mobile devices. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 1–16. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_1
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Wei, K., Fu, Y., Yang, J., Huang, H.: A physics-based noise formation model for extreme low-light raw denoising. In: Proceedings CVPR, pp. 2758–2767 (2020)
Whang, J., Delbracio, M., Talebi, H., Saharia, C., Dimakis, A.G., Milanfar, P.: Deblurring via stochastic refinement. In: Proceedings CVPR, pp. 16293–16303 (2022)
Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. Int. J. Comput. Vis. 98(2), 168–186 (2012)
Xia, Z., Perazzi, F., Gharbi, M., Sunkavalli, K., Chakrabarti, A.: Basis prediction networks for effective burst denoising with large kernels. In: Proceedings CVPR, pp. 11844–11853 (2020)
Xing, Y., Qian, Z., Chen, Q.: Invertible image signal processing. In: Proceedings CVPR, pp. 6287–6296 (2021)
Xu, X., Li, M., Sun, W.: Learning deformable kernels for image and video denoising. arXiv preprint arXiv:1904.06903 (2019)
Yoo, J., Ahn, N., Sohn, K.A.: Rethinking data augmentation for image super-resolution: A comprehensive analysis and a new strategy. In: Proceedings CVPR, pp. 8375–8384 (2020)
Yuan, L., Sun, J., Quan, L., Shum, H.Y.: Image deblurring with blurred/noisy image pairs. ACM Trans. Graph. 26(3), 1-es (2007)
Yuan, Y., Su, W., Ma, D.: Efficient dynamic scene deblurring using spatially variant deconvolution network with optical flow guided training. In: Proceedings CVPR, pp. 3555–3564 (2020)
Yue, Z., Yong, H., Zhao, Q., Meng, D., Zhang, L.: Variational denoising network: toward blind noise modeling and removal. Proc. NeurIPS 32, 1690–1701 (2019)
Zamir, S.W., et al.: CycleISP: real image restoration via improved data synthesis. In: Proceedings CVPR, pp. 2696–2705 (2020)
Zamir, S.W., et al.: Learning enriched features for real image restoration and enhancement. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 492–511. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_30
Zamir, S.W., et al.: Multi-stage progressive image restoration. In: Proceedings CVPR, pp. 14821–14831 (2021)
Zhang, B., Jin, S., Xia, Y., Huang, Y., Xiong, Z.: Attention mechanism enhanced kernel prediction networks for denoising of burst images. In: Proceedings ICASSP, pp. 2083–2087 (2020)
Zhang, H., Dai, Y., Li, H., Koniusz, P.: Deep stacked hierarchical multi-patch network for image deblurring. In: Proceedings CVPR, pp. 5978–5986 (2019)
Zhang, J., et al.: Dynamic scene deblurring using spatially variant recurrent neural networks. In: Proceedings CVPR, pp. 2521–2529 (2018)
Zhang, J., Cao, Y., Fang, S., Kang, Y., Wen Chen, C.: Fast haze removal for nighttime image using maximum reflectance prior. In: Proceedings CVPR, pp. 7418–7426 (2017)
Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26(7), 3142–3155 (2017)
Zhang, K., Zuo, W., Zhang, L.: EFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27(9), 4608–4622 (2018)
Zhang, Y., Wang, C., Maybank, S.J., Tao, D.: Exposure trajectory recovery from motion blur. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image restoration. IEEE Trans. Pattern Anal. Mach. Intell. 43(7), 2480–2495 (2020)
Zhu, X., Hu, H., Lin, S., Dai, J.: Deformable convnets v2: more deformable, better results. In: Proceedings CVPR, pp. 9308–9316 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, Y., Xu, Y., Yan, Q., Yang, D., Wang, X., Po, LM. (2022). D2HNet: Joint Denoising and Deblurring with Hierarchical Network for Robust Night Image Restoration. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-20071-7_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20070-0
Online ISBN: 978-3-031-20071-7
eBook Packages: Computer ScienceComputer Science (R0)