Abstract
Deep learning has proven to be an effective mechanism for computer vision tasks, especially for image denoising and burst image denoising. In this paper, we focus on solving the burst image denoising problem and aim to generate a single clean image from a burst of noisy images. We propose to combine the power of block matching and 3D filtering (BM3D) and a convolutional neural network (CNN) for burst image denoising. In particular, we design a CNN with a divide-and-conquer strategy. First, we employ BM3D to preprocess the noisy burst images. Then, the preprocessed images and noisy images are fed separately into two parallel CNN branches. The two branches produce somewhat different results. Finally, we use a light CNN block to combine the two outputs. In addition, we improve the performance by optimizing the two branches using two different constraints: a signal constraint and a noise constraint. One maps a clean signal, and the other maps the noise distribution. In addition, we adopt block matching in the network to avoid frame misalignment. Experimental results on synthetic and real noisy images show that our algorithm is competitive with other algorithms.
摘要
深度学习在计算机视觉领域应用非常成功, 促进了图像降噪和多帧图像降噪领域的快速发展. 本文针对多帧图像降噪问题, 提出一种从多帧噪声图像中恢复清晰图像的方法. 该方法结合BM3D (块匹配和三维滤波, block-matching and 3D filtering) 算法和卷积神经网络 (CNN) 模型完成多帧图像降噪任务. 该CNN模型基于分治法的思想设计. 首先, 用BM3D算法处理带噪声的多帧图像. 然后, 将预处理后的图像和原始噪声图像分别输入CNN模型的两个并行分支. 最后, 用一个轻量级CNN模块融合两个分支的输出得到最终图像估计. 与以往研究不同, 我们对CNN中两个并行分支分配了不同约束函数——信号约束和噪声约束, 以提升模型提取不同特征的能力. 此外, 引入图像块匹配策略解决帧不对齐问题. 在合成和真实噪声图像上的实验结果表明, 该算法与其他算法相比具有一定竞争力.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aharon M, Elad M, Bruckstein A, 2006. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process, 54(11):4311–4322. https://doi.org/10.1109/TSP.2006.881199
Ahn B, Cho NI, 2017. Block-matching convolutional neural network for image denoising. https://arxiv.org/abs/1704.00524
Buades A, Coll B, Morel JM, 2005. A non-local algorithm for image denoising. IEEE Computer Society Conf on Computer Vision and Pattern Recognition, p.60–65. https://doi.org/10.1109/CVPR.2005.38
Burger HC, Schuler CJ, Harmeling S, 2012. Image denoising: can plain neural networks compete with BM3D? IEEE Conf on Computer Vision and Pattern Recognition, p.2392–2399. https://doi.org/10.1109/CVPR.2012.6247952
Chambolle A, 2004. An algorithm for total variation minimization and applications. J Math Imag Vis, 20(1–2):89–97. https://doi.org/10.1023/B:JMIV.0000011325.36760.1e
Dabov K, Foi A, Katkovnik V, et al., 2007. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans Image Process, 16(8):2080–2095. https://doi.org/10.1109/TIP.2007.901238
Divakar N, Babu RV, 2017. Image denoising via CNNs: an adversarial approach. Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops, p.1076–1083. https://doi.org/10.1109/CVPRW.2017.145
Godard C, Matzen K, Uyttendaele M, 2018. Deep burst denoising. Proc European Conf on Computer Vision, p.560–577. https://doi.org/10.1007/978-3-030-01267-0_33
Krull A, Buchholz TO, Jug F, 2019. Noise2Void—learning denoising from single noisy images. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.2124–2132. https://doi.org/10.1109/CVPR.2019.00223
LeCun Y, Bottou L, Bengio Y, et al., 1998. Gradient-based learning applied to document recognition. Proc IEEE, 86(11):2278–2324. https://doi.org/10.1109/5.726791
Lehtinen J, Munkberg J, Hasselgren J, et al., 2018. Noise2Noise: learning image restoration without clean data. https://arxiv.org/abs/1803.04189
Lempitsky V, Vedaldi A, Ulyanov D, 2018. Deep image prior. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.9446–9454. https://doi.org/10.1109/CVPR.2018.00984
Liu ZW, Yuan L, Tang XO, et al., 2014. Fast burst images denoising. ACM Trans Graph, 33(6):Article 232. https://doi.org/10.1145/2661229.2661277
Mao XJ, Shen CH, Yang YB, 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. https://arxiv.org/abs/1603.09056v2
Mildenhall B, Barron JT, Chen JW, et al., 2018. Burst denoising with kernel prediction networks. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.2502–2510. https://doi.org/10.1109/CVPR.2018.00265
Mosseri I, Zontak M, Irani M, 2013. Combining the power of internal and external denoising. IEEE Int Conf on Computational Photography, p.1–9. https://doi.org/10.1109/ICCPhot.2013.6528298
Perona P, Malik J, 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans Patt Anal Mach Intell, 12(7):629–639. https://doi.org/10.1109/34.56205
Simonyan K, Zisserman A, 2014. Very deep convolutional networks for large-scale image recognition. https://arxiv.org/abs/1409.1556v4
Tassano M, Delon J, Veit T, 2019. DVDNET: a fast network for deep video denoising. IEEE Int Conf on Image Processing, p.1805–1809. https://doi.org/10.1109/ICIP.2019.8803136
Tomasi C, Manduchi R, 1998. Bilateral filtering for gray and color images. Sixth Int Conf on Computer Vision, p.839–846. https://doi.org/10.1109/ICCV.1998.710815
Vincent P, Larochelle H, Lajoie I, et al., 2010. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res, 11:3371–3408.
Xu J, Zhang L, Zuo WM, et al., 2015. Patch group based non-local self-similarity prior learning for image denoising. Proc IEEE Int Conf on Computer Vision, p.244–252. https://doi.org/10.1109/ICCV.2015.36
Yang D, Sun J, 2018. BM3D-Net: a convolutional neural network for transform-domain collaborative filtering. IEEE Signal Process Lett, 25(1):55–59. https://doi.org/10.1109/LSP.2017.2768660
Zhang K, Zuo WM, Chen YJ, et al., 2017. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans Image Process, 26(7):3142–3155. https://doi.org/10.1109/TIP.2017.2662206
Zhang K, Zuo WM, Zhang L, 2018. FFDNet: toward a fast and flexible solution for CNN-based image denoising. IEEE Trans Image Process, 27(9):4608–4622. https://doi.org/10.1109/TIP.2018.2839891
Author information
Authors and Affiliations
Contributions
Dan ZHANG and Lei ZHAO designed the research. Dan ZHANG conducted the experiments and drafted the paper. Lei ZHAO helped organize the paper. Duanqing XU and Dongming LU revised and finalized the paper.
Corresponding author
Ethics declarations
Dan ZHANG, Lei ZHAO, Duanqing XU, and Dongming LU declare that they have no conflict of interest.
Additional information
Project supported by the National Major Program of Key Technology Research of Database Construction and Intelligent Retrieval of Culture Relics Prohibited from Trading, China (No. 2020YFC1523202), the National Major Social Science Fund of Collation and Comprehensive Study of Bronze Ware Data in HAN Dynasty, China (No. 19ZDA197), the Zhejiang Fund of Ancient Painting Image Restoration Based on Prior Background Knowledge Constraints, China (No. LY21F020005), and the Key Scientific Research Base for Digital Conservation of Cave Temples (Zhejiang University), the State Administration for Cultural Heritage, China
Rights and permissions
About this article
Cite this article
Zhang, D., Zhao, L., Xu, D. et al. Dual-constraint burst image denoising method. Front Inform Technol Electron Eng 23, 220–233 (2022). https://doi.org/10.1631/FITEE.2000353
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2000353