Introduction

In real scenes, images are affected by noise and blur, which can lead to a decrease in image quality [1]. The process of image degradation [2] can usually be described as

$$\begin{aligned} y = Ax + n \end{aligned}$$
(1)

where A represents the blur matrix, n represents the noise. In order to improve image quality, model-based methods [3,4,5,6] and learning-based methods [7,8,9,10] are widely used in image restoration problems.

In model-based restoration methods, the objective function contains data fidelity term and regularization term [11]. The data fidelity term ensures consistency between data and makes results consistent with degradation processes. Regularization terms introduce prior information into the model to accomplish specific tasks. Model-based image restoration methods can generally be expressed as the following minimized energy function [12]:

$$\begin{aligned} \widehat{x} = \arg \mathop {\min }\limits _x L(x,y) + \lambda \Psi (x) \end{aligned}$$
(2)

where x represents high-quality images and y represents degraded images.

In Eq. (2), the data fidelity term is used to evaluate the similarity between the observed data and the estimated high-quality image based on the observation model. In recent years, many new fidelity terms [13,14,15,16,17,18,19] have been proposed, and achieves very well results. As we know, the image restoration problem is highly ill-posed. Thus, It is necessary to make reasonable use of prior knowledge. In this paper, we will focus on the study of the prior model to restrict the solution space, improving the reconstruction quality.

The regularization term [20] is used to model prior knowledge and extract important feature information of the image. And it can also punish solutions that do not meet assumptions and constrain the estimated reconstructed image. In [21], Zhou et al. proposed a model with MR image reconstruction incorporating the concept of TV regularization. In [22], Zhang et al. developed an iterative algorithm that utilizes an optimizer to improve image reconstruction and effectively reduce artifacts in image restoration. In [23], Sun et al. proposed a fractional Tikhonov frame imaging method, which has an automatic parameter selection function to ensure image reconstruction accuracy. In [24], Ghulyani et al. designed a new regularization term and constructed a novel image reconstruction model. Zhao et al. [25] proposed a method of using wavelet sparsity-constrained image prior. This was followed by Bhadra et al. [26], who applied adaptive priors, and Gong et al. [27], who applied non-local priors to image reconstruction. However, the regularization term constructed by the above method has subjective tendency and cannot achieve the best effect in restoring image detail information.

The above prior models have achieved some results in improving image quality. However, these prior models are artificially designed, and the subjective tendencies would suppress some image information, leading to poor restoration performance.

Deep learning methods are based on end-to-end mapping relationships. The optimal weight constraint is constructed to gain prior knowledge for image restoration. Up to now, many learning-based methods [28,29,30,31,32] have been proposed to improve image quality, such as Shi et al. [28] proposed a real-world character restoration network, Mei et al. [29] proposed a pyramid attention network, and Ye et al. [31] proposed a network called UIR-Net. These methods could obtain image with high quality, but they cannot effectively utilize the prior information inside the image, and there are still artifacts in the reconstructed image.

To sum up, the model-based methods are flexible to handle inverse problems with complex degradation, but require devising complex priors to achieve good performance. Learning-based methods have been able to learn end-to-end mapping between high-quality and low-quality images, but they cannot effectively utilize the prior information inside the image. Therefore, this paper proposes a method that can simultaneously utilize the advantages of both methods to solve the problem of image restoration.

Hybrid regularization inspired by total variation and deep denoiser prior for image restoration method is proposed in this paper. Based on the Plug-and-Play framework, using both the advantages of TV prior in protecting image edges and the advantages of learned prior in protecting image textures. And set an adaptive parameter to balance the weights of TV prior and learned prior. Contributions to this paper are outlined below:

(1) Constructing an image restoration model based on hybrid prior with adaptive parameter under the plug-and-play framework. Through this framework, the hand-crafted prior and the learned prior are combined to jointly constrain the reconstruction process to effectively restore image details.

(2) Introducing TV prior and fast and flexible denoising convolutional neural network (FFDNet) networks, utilizing the advantages of TV prior being able to protect image edge information and the advantages of learned prior being able to protect image texture, allowing both to jointly protect image detail information.

(3) To balance the weights of TV prior and the learned prior, a function based on the gradient information is designed to estimate the weight parameter adaptively. For images with sharp outline, increase the contribution of TV. Otherwise, increase the contribution of the learned prior model.

The remaining parts of this paper are as follows: Section “Materials and Methods” introduces the method proposed in this paper, section “Experiment” demonstrates the effectiveness of the proposed method through experiments, and section “Conclusion” provides a brief conclusion.

Materials and Methods

Hybrid regularization inspired by the total variation prior and deep denoiser prior for image restoration method (called HRIR for short) is proposed in this paper. In HRIR, the TV prior and learned prior are introduced to protect image edge and texture information, respectively. Moreover, a gradient based method is designed to adaptively estimate the weight parameter which could balance these two models.

Proposed reconstruction model

Assuming that the observed image y degrades from the potential high-quality image x. To obtain high-quality images, the TV based restoration model is established as follows:

$$\begin{aligned} \hat{x} = \arg \mathop {\min }\limits _x \frac{1}{2}||y - Ax||_2^2 + \lambda \Phi (x) \end{aligned}$$
(3)

where A is the degradation matrix; \(\lambda \) is regularization parameter.

In Eq. (3), \(||y - Ax||_2^2\) is the data fidelity term that measures the similarity of y and Ax. \(\Phi (x)\) is the regularization term. Here, in order to preserve the edge details, the TV model is introduced into Eq. (3), i.e., \(\Phi (x) = ||Dx|{|_1}\), where D represents the gradient operator. As we know, the TV model usually results in over-smoothing, which is not conducive to protecting texture details. Thus, another regularization term J(x) is adopted in the restoration model. Here, J(x) represents the prior information introduced by the deep denoising network. Such a model is designed to provide more information for recovering more realistic texture details. Therefore, \(\Phi (x)\) can be expressed as: \(\Phi (x)=\alpha J(x) + (1 - \alpha )||Dx|{|_1}\).

Based on the above analysis, a new reconstruction model based on hybrid priors is established as follows,

$$\begin{aligned} \hat{x} = \arg \mathop {\min }\limits _x \frac{1}{2}||y - Ax||_2^2 +\lambda [\alpha J(x) + (1 - \alpha )||Dx|{|_1}]\nonumber \\ \end{aligned}$$
(4)

where A is the degradation matrix; \(\lambda \) and \(\alpha \) are regularization parameters. Where \(\lambda \) is a fixed constant related to noise level, and \(\alpha \) is an adaptive parameter updated in each iteration. Use \(\lambda \) to better constrain the reconstruction process, and use \(\alpha \) to balance the weights of the two models.

The optimization

Since Eq. (4) is an unconstrained function, it is difficult to solve it directly. Therefore, this paper considers introducing two auxiliary variables and iteratively solving Eq. (4) into three sub problems using the HQS method, as shown in HRIR algorithm.

HRIR algorithm

input :  The low quality image y, maximum iteration N, \(\alpha ,\lambda \), \(\gamma _1\), \(\gamma _2\);

output :  The high quality image x;

1. \(k=0\);

2. Use low quality image y to represent the initial image \({x^0}\);

3. Calculate the degradation matrix A;

4. \({z_1{\mathrm{{^0}}}} = {x^0},{z_2{\mathrm{{ ^0}}}} = D{x^0}\);

5. for \(k=0:N\)

6. Update \(\alpha , \gamma _1\), \(\gamma _2\);

7. Calculate \(x^{k + 1}\)using equation (7a);

8. Calculate \(z_1^{k + 1}\)using equation (7b);

9. Calculate \(z_2^{k + 1}\)using equation (7c);

10. \(x = x^{k + 1}\);

11. \(k=k+1\);

12. end for

First, two auxiliary variables \({z_1},{z_2}\) are introduced. Thus, Eq. (4) could be rewritten as follows,

$$\begin{aligned} (\hat{x},\widehat{{z_1}},\widehat{{z_2}})= & {} \arg \mathop {\min }\limits _{x,{z_1},{z_2}} \frac{1}{2}||y - Ax||_2^2 + \lambda \alpha J({z_1})\nonumber \\{} & {} + \lambda (1 - \alpha )||{z_2}||_1^{}\quad s.t.\quad {z_1} = x,{z_2} = Dx \end{aligned}$$
(5)

The augmented lagrangian function corresponding to Eq. (5) is:

$$\begin{aligned} (\hat{x},\widehat{{z_1}},\widehat{{z_2}})= & {} \arg \mathop {\min }\limits _{x,{z_1},{z_2}} \frac{1}{2}||y - Ax||_2^2 \nonumber \\{} & {} + \lambda \alpha J({z_1}) + \lambda (1 - \alpha )||{z_2}||_1^{}\mathrm{{ + }}\frac{{{\gamma _1}}}{2}||{z_1} - x||_2^2\nonumber \\{} & {} \mathrm{{ + }}\frac{{{\gamma _2}}}{2}||{z_2} - Dx||_2^2 \end{aligned}$$
(6)

where \(\gamma _1\), \(\gamma _2\) are penalty parameters for iterative changes in non-descending order. \(\gamma _1\), \(\gamma _2\) update with iteration.

Then, Eq. (6) is split into the following subproblems,

figure a

In Eqs. (7a) to (7c), introducing the auxiliary variable \({z_1},{z_2}\) could separate the data fidelity term and the regularization term into subproblems on \(x,{z_{1}},{z_2}\), respectively. Figure 1 shows the main steps of the algorithm.

Fig. 1
figure 1

Algorithm flowchart the HRIR method

Full size image
Fig. 2
figure 2

FFDNet structure

Full size image

(a)Solving the x-subproblem

Subproblem Eq. (7a) is related to the data fidelity item \(||y - Ax||_2^2\), which is a restoration problem constrained by the prior model based on the \(L_2\) norm. Thus, Eq. (7a) could be solved by \(\frac{{\partial {F_1}}}{{\partial x}} = 0\), where \({F_1}(x) = \frac{1}{2}||y - Ax||_2^2 + \frac{{{\gamma _1}}}{2}||{z_1^k} - x||_2^2 + \frac{{{\gamma _2}}}{2}||{z_2^k} - Dx||_2^2\), and the following expression is obtained,

$$\begin{aligned} ({A^T}A + {\gamma _1}I + {\gamma _2}{D^T}D){x^{k + 1}} = ({A^T}y + {\gamma _1}{z_1^k} + {\gamma _2}{D^T}{z_2^k})\nonumber \\ \end{aligned}$$
(8)

Then, the fast Fourier transform (FFT) is applied to solve Eq. (8), and \(x^{k + 1}\) would be calculated by the following formula,

$$\begin{aligned} {x^{k + 1}} = {F^{ - 1}}\left( \frac{{\overline{F(A)} F(y) + {\gamma _1}F({z_1^k}) + {\gamma _2}\overline{F(D)} F({z_2^k})}}{{\overline{F(A)} F(A) + {\gamma _1}I + {\gamma _2}\overline{F(D)} F(D)}}\right) \end{aligned}$$
(9)

where \(F(\bullet ){\text { represents the Fourier transform, and }}{F^{ - 1}}(\bullet )\) represents the inverse Fourier transform. \(\overline{F(\bullet )}\) is a conjugate complex transformation of the Fourier transform.

(b)Solving the \({z_1}\) -subproblem

Subproblem Eq. (7b) is related to the regularization term J(x). This subproblem could be seen as a denoising problem, \(x = {z_1} + n\), where n represents noise.

Since solving the prior model is a complex computational process, consider using a learning-based method instead of solving the prior model, updating \(z_{1}\) by using a denoising algorithm on x. Eq. (7b) could also be rewritten as:

$$\begin{aligned} {z_1^{k + 1}} = Denoiser({x^{k + 1}},\sqrt{{\lambda \alpha / {{\gamma _1}}}} ) \end{aligned}$$
(10)

From subproblems Eqs. (7b) and (10), it is shown that a denoising network could be introduced to address the problem presented in Eq. (7b). Thus, the prior model J(x) is replaced by the denoising network, which avoids the design of complex regularization models. Then subproblem Eq. (7b) could be solved directly using an off-the-shelf denoiser.

In this paper, the network, FFDNet [33], is considered to deal with the denoising problem, which could handle a variety of noise levels. As shown in Fig. 2, the FFDNet introduces a downsampling operator with reversible that reshapes one noise image into several downsampled subimages. A noise level map M is then connected with the subimage to form a tensor \(\tilde{y}\). The tensor \(\tilde{y}\) is input to the convolutional layer for feature extraction. To maintain a constant size of the feature map after each convolution, zero-padding is used. Following the final layer, an upsampling operation is performed to derive a clear image.

(c)Solving the \({z_2}\) -subproblem

Subproblem Eq. (7c) is related to the regularization term \(||Dx|{|_1}\). This subproblem could be seen that establishing \({z_2}\) from the noisy data Dx, and such a problem could be described as \(Dx = {z_2} + \varepsilon \), where \(\varepsilon \) represents noise.

The objective function in Eq. (7c) is based on the \(L_1\) norm, the iterative shrinkage threshold algorithm is introduced to address the denoising problem. By using the soft threshold operator, the subproblem Eq. (7c) could be solved as:

$$\begin{aligned} z_{2\mathrm{{ }}}^{k + 1} = sof{t_{\frac{\lambda (1-\alpha ) }{{{\gamma _2}}}}}(Dx^{k+1}) \end{aligned}$$
(11)

with

$$\begin{aligned} sof{t_{\frac{\lambda (1-\alpha ) }{{{\gamma _2}}}}}(t) = \left\{ \begin{array}{l} t + \frac{\lambda (1-\alpha ) }{{{\gamma _2}}},\mathrm{{ }}\hspace{5.0pt}t< - \frac{\lambda (1-\alpha ) }{{{\gamma _2}}};\\ 0,\quad \quad \quad \quad \hspace{5.0pt}\mathrm{{ | }}t\mathrm{{ | < }}\frac{\lambda (1-\alpha ) }{{{\gamma _2}}};\\ t - \frac{\lambda (1-\alpha )}{{{\gamma _2}}},\quad \mathrm{{ }}t > \frac{\lambda (1-\alpha ) }{{{\gamma _2}}}; \end{array} \right. \end{aligned}$$

Adaptive parameter estimating

In Eq. (4), in order to better make use of the TV prior and learned prior, the parameter estimation function based on the gradient information is set here to balance these two models adaptively. Shown as follows:

$$\begin{aligned} \alpha = \frac{1}{{(l \times mean(N(\root 2 \of {{|Gx|}}))) + 1}} \end{aligned}$$
(12)

where N(x) is represented as the normalization function, G represents gradient operator, and l is the weight coefficient, with \(l=7\).

Equation (12) shows the parameter estimation function based on the gradient information. When \(|Gx| \rightarrow 0\), it indicates a weak edge, and when \(|Gx| \rightarrow \infty \), it represents a strong edge. Therefore, we use gradient-based functions to estimate the adaptive parameter. In this model, as \(\alpha \) is inversely proportional to the gradient, the reciprocal of the gradient is used as a function to represent \(\alpha \). In addition, to ensure that the denominator is not 0, a constant 1 is added to the denominator. In the implementation process, considering the order of magnitude of the gradient, we use the square root of the gradient to replace the gradient of the image.

Experiment

In this section, a comparison is made between our proposed method and several image restoration techniques, including FFDNet [33], CBDNet [34], weighted nuclear norm minimization (WNNM) [10], image restoration convolutional neural network (IRCNN) [35], denoiser prior image restoration (DPIR) [36]. In our paper, several objective and subjective indicators were used to evaluate the effect. The results of these experiments are presented in the following subsections.

Experiment preparation

We have assembled a collection of source images for our training dataset, which includes 400 images from the BSD dataset and 5000 images from the Paris dataset. We use these datasets (Set5, Set12, Set18, and Set24) to validate the proposed algorithm performance.

Table 1 Different levels of noise and blur
Full size table
Table 2 PSNR/SSIM obtained with different values of parameter "l" under different noise and blur conditions
Full size table
Table 3 PSNR/SSIM values of each module under different noise and blur
Full size table
Table 4 PSNR (dB) and SSIM of various methods under Gaussian noise with the level of 0.01 and the 3\(\times \)3 Gaussian blur kernel
Full size table
Table 5 PSNR (dB) and SSIM of various methods under Gaussian noise with the level of 0.03 and the 5\(\times \)5 Gaussian blur kernel
Full size table
Table 6 PSNR (dB) and SSIM of various methods under Gaussian noise with the level of 0.01 and the 3\(\times \)3 Average blur kernel
Full size table
Table 7 PSNR (dB) and SSIM of various methods under Gaussian noise with the level of 0.03 and the 5\(\times \)5 Average blur kernel
Full size table

In this paper, we use subjective effects and objective indicators as the evaluation criteria for experimental results. Subjective effects reflect the reconstruction effects of different methods by reconstructing images. We use Peak Signal to Noise Ratio (PSNR) (dB) and Structural Similarity (SSIM) as objective indicators for evaluation. The larger the PSNR and SSIM, the better the effect. Eqs. (13) and (14) respectively provide the calculation formulas for PSNR and SSIM.

$$\begin{aligned}{} & {} PSNR = 10 \cdot {\log _{10}}\frac{{{{(MAX_I^{})}^2}}}{{\frac{1}{{mn}}\sum \nolimits _{i = 0}^{m - 1} {\sum \nolimits _{j = 0}^{n - 1} {{{[I(i,j) - K(i,j)]}^2}} } }} \end{aligned}$$
(13)
$$\begin{aligned}{} & {} SSIM(I,K) = \frac{{(2{\mu _I}{\mu _K} + {c_1})(2{\sigma _{IK}} + {c_2})}}{{(\mu _I^2 + \mu _K^2 + {c_1})(\sigma _I^2 + \sigma _K^2 + {c_2})}} \end{aligned}$$
(14)

In Eqs. (13) and (14), I represents original image, K represents reconstruction image, \(m \times n\) is the size of the image I, K. (i, j) represents the pixel points in images I, K. \(MA{X_I}\) is the maximum possible pixel value of the original image. \({\mu _I}\) is the mean of I; \({\mu _K}\) is the mean of K; \(\sigma _I^2\) is the variance of I; \(\sigma _K^2\) is the variance of K; \({\sigma _{IK}}\) is the covariance of I and K. \({c_1}\) and \({c_2}\) are constant.

To generate low-quality images, different levels of noise and blur are added to the datasets. The images are then used in experiments. Table 1 presents the noise and blur with different levels used in this paper.

We also set the parameters in the model. For the reconstruction process, \(\lambda \) is related to noise level, and remain fixed in each iteration. For penalty parameters \(\gamma _1\) and \(\gamma _2\), set an interval of [7,49], select multiple points within this interval, and the number of points is equal to the number of iterations. The value of each point decreases logarithmically. Set a function based on noise level and selected points to represent parameters \(\gamma _1\) and \(\gamma _2\).

In this paper, all methods produce results generated by code provided by the authors and set the parameters of their experiments to the optimum value.

Ablation study

This section first tested the value of parameter l in Eq. (12), and we conducted a series of experiments on the Set5 dataset to select the value of parameter l under different noise and blur environments. The experimental results are conducted with different values of l from 0.1 to 20. The interval between the values of each point is 0.1. Part of the experimental results are presented in the following Table  2. The experimental results have shown that the reconstructed image obtained is the best when l=7.

In addition, this paper also carried out ablation experiments to verify the effectiveness of each module of the proposed method. We will remove the TV model and FFDNet model separately and conduct experiments on the remaining modules. Experiments were conducted on images in different noise and blur environments using the Set5 dataset, and the experimental results are shown in Table 3.

Experimental results demonstrate the effectiveness of each module of this method, and each module combination achieves optimal PSNR (dB) and SSIM.

Fig. 3
figure 3

Image recovery results of Gaussian noise with the level of 0.01 and 3\(\times \)3 Gaussian blur kernel. a Original image (PSNR (dB)/SSIM), b HRIR (33.90/0.9347), c IRCNN (33.89/0.9010), d FFDNet (26.86/0.9125), e CBDNet (26.81/0.9161), f DPIR (33.56/0.9154), g WNNM (26.83/0.9179)

Full size image

Experimental results

In this section, the effectiveness of our method is fully demonstrated through objective indicators and subjective effects. First, this paper adds different Gaussian blur kernels (a 3\(\times \)3 Gaussian blur kernel with variance of 1 and a 5\(\times \)5 Gaussian blur kernel with variance of 1.6) to the test images. In addition, Gaussian noise with noise levels of 0.01 and 0.03 is added to generate low-quality images. Tables 4 and 5 show the results of the average PSNR (dB) and average SSIM under different Gaussian noise levels and Gaussian blur kernel conditions. It can be seen from these two tables that in the situation of Gaussian white noise with different noise levels and Gaussian blur, our method could obtain better average PSNR (dB) and average SSIM. These results indicate the superior performance of the proposed method discussed in this paper.

In addition, we also conducted experiments under different sizes of Average blur kernels and different Gaussian noise levels. Objective indicators are shown in Tables 6 and 7. We use Gaussian noise with the level of 0.01 and 3\(\times \)3 Average blur kernel as well as Gaussian noise with the level of 0.03 and 5\(\times \)5 Average blur kernel. Under these conditions, the proposed method achieves the optimal average PSNR (dB) and average SSIM results.

To demonstrate the subjective validity of the proposed method, restore the images (Figs. 3, 4and 5) at various noise levels and blur kernels is given. For detailed comparison, the selected portion of the red rectangle in each image is enlarged. It should be noted that the enlarged area could show more detail. As shown in Fig. 3, the method proposed in this paper could preserve the image details more effectively, and the texture part of the restored image is clearer. For the restoration of the“Baby face”image and the“Butterfly”image (shown in Figs. 4 and 5, respectively), our method obtains the result with less artifacts and clearer edges, providing a better view. Our proposed method protects the image details by introducing the TV and learned prior and achieves the best visual effects.

Through the above experimental results, it has been proved that the method proposed in this paper outperforms the comparable method in both objective indicators and subjective effects. Therefore, the effectiveness of the proposed method has been demonstrated. Based on the Plug-and-Play framework, combine TV prior and learned prior, and set adaptive parameters to balance weights of the two. The new reconstruction model constructed in this way has good results in image restoration.

Fig. 4
figure 4

Image recovery results of Gaussian noise with the level of 0.03 and 5\(\times \)5 Gaussian blur kernel. a Original image (PSNR (dB)/SSIM), b HRIR (32.98/0.8674), c IRCNN (31.77/0.8003), d FFDNet (30.59/0.8547), e CBDNet (30.64/0.8554), f DPIR (31.97/0.8672), g WNNM (30.62/0.8565)

Full size image
Fig. 5
figure 5

Image recovery results of Gaussian noise with the level of 0.01 and 3\(\times \)3 Average blur kernel. a Original image (PSNR (dB)/SSIM), b HRIR (33.74/0.9559), c IRCNN (32.61/0.9138), d FFDNet (26.96/0.9163), e CBDNet (25.39/0.8760), f DPIR (25.44/0.8846), g WNNM (25.64/0.8942)

Full size image

Conclusion

In this paper, hybrid regularization inspired by total variation and deep denoiser prior for image restoration method is proposed. Using the plug-and-play framework to combine TV prior and learned prior, a new reconstruction model is constructed. At the same time, the advantages of TV prior protecting image edges and learned prior protecting image textures are used to restore image details more effectively. In addition, in order to better balance the weights of TV prior and learned prior, a function based on gradient is set as adaptive parameter to automatically balance the weights of the two in each iteration. Experimental results show that our method achieved optimal results.