1 Introduction and contributions

Reverse image filtering is a recent and promising research direction in image processing [2, 9, 20, 30]. Given an image filter, the goal of reverse image filtering consists of removing or suppressing of the filter effects using only the filter itself. Furthermore, the filter internal structure is not supposed to be known, the filter is allowed to be used as many times as needed but only as a black box.

The idea of multiple use of the same filter for designing a new filter or improving existing filter properties is not new, of course. It goes back to Tukey’s twicing scheme [12, 32] and was further developed by Kaiser and Hamming [14]. See also an expository paper [8]. Similar approaches have been recently proposed in [23, 25]. Reverse image filtering is a related but different problem, as our aim is to invert the filter, not to enhance filter properties.

In spite that the reverse image filtering idea was introduced very recently [30], it has already contributed to developing new deep learning architectures [6], studies on iterative depth and image super-resolution [22, 35], and image restoration and sharpening [7, 28].

The problem setting for reverse image filtering is simple and consists of restoring an input image \({{\varvec{x}}}\) from the observed output image \({{\varvec{y}}}= {{\varvec{f}}}\,({{\varvec{x}}})\) where \({{\varvec{f}}}\,(\cdot )\) is an image filter. It can be formulated as an inverse problem \({{\varvec{x}}}={{\varvec{f}}}\,^{-1}({{\varvec{y}}})\). We assume that filter \({{\varvec{f}}}\,(\cdot )\) is only available to us as a black-box, i.e., it can be queried, but it cannot be directly inverted, as we don’t know its internal structure. Thus, the goal of reverse filtering consists of estimating the filter inverse \({{\varvec{f}}}\,^{-1}(\cdot )\) using iterations of the filter \({{\varvec{f}}}\,(\cdot )\) itself.

The previous studies on reverse image filtering [2, 9, 20, 30] mostly work with symmetric and clean filters. However, in real-life scenarios, we often deal with non-symmetric filters (e.g., motion blur filters) which can also be contaminated by noise. In Fig. 1, we present a simple experiment with the convolutional filter defined by

$$\begin{aligned} {{\varvec{f}}}\,({{\varvec{x}}})\equiv {{\varvec{h}}}\circledast {{\varvec{x}}}+{{\varvec{n}}}, \end{aligned}$$
(1)

where \({{\varvec{h}}}\) is a motion blur kernel and \({{\varvec{n}}}\) is a zero-mean Gaussian noise. The reader can see that both the reverse filtering methods of [30] and [9] fail to achieve a satisfactory reconstruction of original image \({{\varvec{x}}}\): the method of [30] does not cope with non-symmetric filters (the same can be said about the schemes proposed in [20] and [2]) and the method of [9] demonstrates a noise-amplification behavior. In contrast, a Landweber-like iteration scheme proposed in this paper yields a very satisfactory reconstruction of the original image.

Fig. 1
figure 1

a Original Lena image. b Motion blurring kernel h used in (1) is shown. c Blurred and noisy image generated by (1) where \({{\varvec{n}}}\) is additive Gaussian noise with zero mean and variance equal to \(10^{-4}\). d The method of Tao et al. [30] is unable to cope with non-symmetric filters. e The method of Dong et al. [9] amplifies noise. f A phase-corrected Landweber-like iteration scheme (to be discussed later in the paper) yields a satisfactory reconstruction of the original image

Full size image

Despite their seemingly simple nature, the Landweber iteration method and its relatives constitute a family of powerful and popular image restoration tools [3, 27].

Our approach to reverse image filtering is based on combining Landweber-like iterations with an iterative convolution-based approximation of a given filter. We demonstrate advantages of the approach over the existing reverse image filtering methods and consider its applications to image deblurring and super-resolution. We demonstrate that total variation regularization, Nesterov acceleration and some image processing techniques can be incorporated in our reverse image filtering framework.

2 Previous work

Mathematically we deal with the equation

$$\begin{aligned} {{\varvec{y}}}={{\varvec{f}}}\,({{\varvec{x}}}) \quad {{\varvec{x}}},{{\varvec{y}}}\in {\mathbb {R}}^d, \end{aligned}$$
(2)

where \({{\varvec{y}}}\) is an observed filtered image, \({{\varvec{f}}}\,(\cdot )\) is a given black-box filter, and the goal is to achieve an accurate estimation of the original image \({{\varvec{x}}}\).

At the first glance, solving (2) numerically seems to be a simple task. For example, estimating the gradient (the Jacobian matrix) of \({{\varvec{f}}}\,({{\varvec{x}}})\) allows us to use standard equation solving methods including Newton’s iterations.

The problem here is that even for a very small \(100\times 100\) image, the Jacobian matrix has size of \(10^4\times 10^4\) and that makes gradient-based methods computationally infeasible.

This calls for developing and using gradient-free methods for numerically solving (2). Gradient-free methods, also known as zeroth-order optimization methods, are currently a subject of intensive research in connection to their applications in Machine Learning [18]. Below we review gradient-free methods introduced for solving (2) in the context of reverse image filtering.

Zero-order Reverse Filtering. Tao et al. [30] propose the following simple iterative reverse filtering scheme

$$\begin{aligned} {{\varvec{x}}}_{k+1}={{\varvec{x}}}_k+{{\varvec{y}}}-{{\varvec{f}}}\,({{\varvec{x}}}_k), \end{aligned}$$
(3)

where \({{\varvec{x}}}_0={{\varvec{y}}}\). The term \({{\varvec{y}}}-{{\varvec{f}}}\,({{\varvec{y}}})\) contains details of \({{\varvec{y}}}\) removed by filter \({{\varvec{f}}}\,(\cdot )\). The idea is to iteratively use this residual to approximate the lost details of \({{\varvec{x}}}\) and add them back to estimate the original version. The whole procedure can be considered as an iterative version of unsharp masking, a simple and popular image sharpening tool.

The authors of [30] consider (3) as a fixed-point map and establish its convergence for linear filters satisfying certain conditions. In practice, (3) shows a very good performance for a number of popular nonlinear image filters. However, as demonstrated in Fig.1, (3) fails to invert non-symmetric filters including linear motion blur filters.

Rendition. A generalization of (3) proposed by Milanfar [20] consists of running the following iterative process

$$\begin{aligned} {{\varvec{x}}}_{k+1}=(1-\gamma \tau ){{\varvec{x}}}_k+\gamma ({{\varvec{y}}}- {{\varvec{f}}}\,({{\varvec{x}}}_k)), \end{aligned}$$
(4)

where \(\gamma \) and \(\tau \) are user-specified parameters. In theory, (4) has better convergence properties than (3). In practice, however, both methods demonstrate similar performance.

First-order Reverse Filtering. Dong et al. [9] propose a gradient-free defiltering scheme which they call an iterative first-order reverse image filtering method, as Newton’s iterations are used [9] to derive the method. The method performs its iterations in the frequency domain

$$\begin{aligned} X_{k+1}=Y/H_k, \quad \mathrm {with}\quad H_k=F({{\varvec{x}}}_k)/X_k, \end{aligned}$$
(5)

where point-wise division is used and where, here and everywhere below, the capital letters indicate the Fourier transforms of the images denoted by the corresponding lower case letters. For example, \(F({{\varvec{x}}}_k) = {\mathcal {F}}\{{{\varvec{f}}}\,({{\varvec{x}}}_k)\}\) and \(X_k={\mathcal {F}}\{{{\varvec{x}}}_k\}\), where \({\mathcal {F}}\{\cdot \}\) denotes the Fourier transform.

We can interpret (5) as follows. If \({{\varvec{f}}}\,(\cdot )\) is a linear convolution filter

$$\begin{aligned} {{\varvec{f}}}\,({{\varvec{x}}})={{\varvec{h}}}\circledast {{\varvec{x}}}, \end{aligned}$$
(6)

then its corresponding frequency response function is given by

$$\begin{aligned} H=F({{\varvec{x}}})/X. \end{aligned}$$
(7)

Thus, in each iteration, (5) approximates \({{\varvec{f}}}\,(\cdot )\) by a linear convolution.

The method of Dong et al. demonstrates excellent performance when dealing with linear filters but is very sensitive to boundary conditions (see examples in [2]) and, in addition, cannot cope with image noise, as shown in Fig.1.

Other reverse image filtering schemes. Finally two reverse filtering methods proposed in [2] are designed to suppress severe filtering effects produced by clean and symmetric image filters and are not appropriate to deal with noisy and/or non-symmetric filters.

3 Proposed approach

The reverse image filtering approach we pursue in this paper is based on using Landweber-like iterations applied to a quadratic error energy function. Namely let us consider the following energy minimization problem

$$\begin{aligned} \Vert {{\varvec{f}}}\,({{\varvec{x}}})-{{\varvec{y}}}\,\Vert ^2\rightarrow \min . \end{aligned}$$
(8)

If \({{\varvec{f}}}\,(\cdot )\) is a linear filter, \({{\varvec{f}}}\,({{\varvec{x}}})=\mathcal{A}{{\varvec{x}}}\), then it seems natural to restore \({{\varvec{x}}}\) by applying the standard Landweber iterations [3, 15]

$$\begin{aligned} {{\varvec{x}}}_{k+1}={{\varvec{x}}}_k-\tau \mathcal{A}^*(\mathcal{A}{{\varvec{x}}}_k-{{\varvec{y}}}), \end{aligned}$$
(9)

where \(\mathcal{A}^*\) is the adjoint (Hermitian conjugate) of operator \(\mathcal{A}\) and \(\tau >0\) is a sufficiently small step-size parameter. Although, for linear \({{\varvec{f}}}\,(\cdot )\), this iterative process represents a simple gradient descent for quadratic energy (8), (9) and its generalizations and modifications remain popular tools for various linear and nonlinear inverse problems including image deconvolution [27].

If filter \({{\varvec{f}}}\,(\cdot )\) is a linear convolution (6), then, due to Parseval’s identity, (8) can be transferred to the frequency domain

$$\begin{aligned} \Vert HX-Y\Vert ^2\rightarrow \min , \end{aligned}$$
(10)

where the transfer function H is defined by (7).

Assume that we deal with a linear convolution filter and H is fixed. Then, in the frequency space, Landweber iterations (9), a special case of gradient descent for (10), are given by

$$\begin{aligned} X_{k+1}=X_k-\tau (H^*(H X_k-Y)), \end{aligned}$$
(11)

where \(H^*\) denotes the complex conjugate of H and \(\tau \) is the step size. Now combining (11) with estimating the transfer function H at each iteration

$$\begin{aligned} H_k=F({{\varvec{x}}}_k)/X_k, \end{aligned}$$
(12)

leads us to

$$\begin{aligned} X_{k+1}=X_k-\tau H_k^*(F({{\varvec{x}}}_k)-Y). \end{aligned}$$
(13)

Note that (13) works in the frequency space and, at each step, approximates nonlinear filter \({{\varvec{f}}}\,(\cdot )\) by a linear convolution filter whose transfer function is determined by (12).

This simple Landweber-like iterative procedure is summarized in Algorithm 1. Similar to the other reverse filtering methods, the algorithm only requires access to the filter as a black-box (without knowledge of its internal structure).

figure a

One advantage of using the variational approach (8) for reverse image filtering is that (8) can easily be extended by adding additional terms incorporating various image priors. Here we are interested in using the total variation prior [26] which leads to a better preservation of salient image edges. Namely, instead of (8) we consider

$$\begin{aligned} \Vert {{\varvec{f}}}\,({{\varvec{x}}})-{{\varvec{y}}}\,\Vert ^2+\gamma \int |\nabla {{\varvec{x}}}|\rightarrow \min , \end{aligned}$$
(14)

where \(\gamma \) is a positive parameter controlling the strength of the prior. Our Landweber-like iterations for (14) are given by

$$\begin{aligned} X_{k+1}\!=\!X_k - \tau H_k^*(F({{\varvec{x}}}_k)-Y) +\tau \gamma {\mathcal {F}}\left\{ \nabla \!\cdot \frac{\nabla {{\varvec{x}}}_k}{|\nabla {{\varvec{x}}}_k | }\right\} . \end{aligned}$$
(15)

The results of our numerical experiments presented in Sect. 4 demonstrate that adding the total variation term improves reverse filtering results for deblurring and super-resolution tasks.

Table 1 Evaluation results in PSNR (dB) for our method compared with the methods of Tao et al. and Dong et al. Linear filter: f = @(x) imfilter(x,H,’circular’); GS: H = fspecial(’gaussian’, [5 5], 2); MT: H = fspecial(’motion’, 21, 11); LOG: H = fspecial(’log’, [5 5], 0.7); DK: H = fspecial(’disk’, 5); UMF: f = @(x) imsharpen(x); GF: f = @(x) imguidedfilter(x); AWF: f = @(x) wiener2(x, [5 5], (10/255)\(\hat{2}\)); L0: f = @(x) L0Smoothing(x, 0.001); BF: f = @(x) bilateralFilter(x,[]); AMF:f = @(x) adaptive_manifold_filter(x, 20, 0.4); LE: f = @(x) localExtrema(x); RF: f = @(x) RF(im2double(x), 30, 0.1); WMF: f = @(x) jointWMF(x); RGF: f = @(x) RollingGuidanceFilter(x); RTV: f = @(x) tsmooth(x); WLS: f = @(x) wlsFilter(x)
Full size table
Fig. 2
figure 2

Comparison results for Gaussian filter (top) and motion blur (bottom). The results of Dong’s method can be considered a lossless restoration of the original image

Full size image
Fig. 3
figure 3

Defiltering example for linear filters and nonlinear filters

Full size image

4 Numerical experiments

4.1 Defiltering

To evaluate the methods developed in this work, we selected 16 popular image filters. The first four, Gaussian filter (GS), motion blur (MT), Laplacian of Gaussian (LOG), and disk filter (DK), are simple linear filters, whose operations on the image can be described by the convolution with a kernel. Except the unsharp masking filter (UMF), the remaining filters are nonlinear. Adaptive Wiener filter (AWF) [19] and weighted median filter (WMF) [34] are used for noise removal. Guided filter (GF) [13], L0 Smoothing (L0) [17], bilateral filter (BF) [31], weighted least squares filter (WLS) [10], domain transform filter (RF) [11], adaptive manifolds filter (AMF) [19], rolling guidance filter (RGF) [24] are used in edge-preserving image smoothing. Local extrema filter (LE) [29] and relative total variation (RTV) [33] are often used for image decomposition purposes.

Table 1 shows reverse filtering PSNR results for the cameraman image (see Fig. 2) for the above-mentioned filters. We compare the proposed method with the methods of Tao et al. [30] and Dong et al. [9]. The parameters for our Landweber-like iteration method (15) are set to \(N=10K\), \(\tau =0.1\), and \(\gamma =0\) (i.e., the total variation regularization is not used).

Fig. 4
figure 4

Successful results of the proposed method for a noisy filter, \(\tau \)=0.1, \(\gamma \)=0.0005 for (f)

Full size image
Fig. 5
figure 5

Deblurring results for noisy filters. Top row: a–e Filtered by imnoise(imfilter(x, fspecial(’gaussian’, [5 5],2),’circular’),’poisson’). \(\tau \) = 2, \(\gamma \) = 0, N = 10000. Bottom row: (f-j) Filtered by imnoise(imfilter(x, testkernel2,’circular,’conv’), ’gaussian,’ 0,0.0001). The kernel is shown in the lower left corner in f. \(\tau = 0.1, \gamma = 0.0005, N=2K\) are used as parameters. Li [16] and Pan [1] are the state-of-the-art blind deblurring methods. Chan [4] is the state-of-the-art non-blind deblurring method. The proposed approach corresponds to iterations of (15)

Full size image

We start from considering the noise-free linear filters from the above list. For them, the Dong et al. method [9] demonstrates the best performance. Our Landweber-like iteration method comes second, as the method of Tao et al. [30] which often fails to reverse linear convolutional filters. As shown in Figs. 2 and 3 a-d, our method allows for recovering most of the details from the original image.

For nonlinear filters, Tao’s method usually performs best, while Dong’s method often has difficulty giving reasonable results, or even fails. In contrast, the proposed method demonstrates always good results, though it does not always deliver the best performance. See Fig. 3 e-h for examples of restored images with the proposed method for nonlinear filters.

4.2 Deblurring

In the previous section, we have demonstrated the effectiveness of the proposed method for reversing Gaussian and motion blur. This motivates us to explore applying the proposed method to the deblurring task. We model a blurred image by

$$\begin{aligned} {{\varvec{y}}}= {{\varvec{h}}}\circledast {{\varvec{x}}}+ {{\varvec{n}}}, \end{aligned}$$
(16)

where \({{\varvec{y}}}\) is the observed blurred image contaminated by noise, \({{\varvec{h}}}\) is a blurring kernel (we assume that \({{\varvec{h}}}\) is not known, but \({{\varvec{f}}}\,(\cdot )\) is available), \(\circledast \) is the convolution operation, \({{\varvec{x}}}\) is the latent image to be recovered, and \({{\varvec{n}}}\) is additive noise. In addition to (16), we also consider the corresponding clean filter obtained from (16) by setting \({{\varvec{n}}}=0\).

As illustrated in Fig. 1, the methods of Tao et al. and Dong et al. [9, 30] do not work well when dealing with noisy filters. In contrast, as seen in Fig. 4 c, d, our approach is robust to noise. In this example, imnoise(imfilter(x,fspecial(’motion’,21,11),’circular’), ’gaussian’,0,0.001) is used as a noisy filter (with MATLAB notation). Using PSNR as a quality measure for defiltering, the best result is obtained at around 850 iterations of our Landweber-like iterations, and subsequent iterations seem to degrade the quality of the image (see Fig. 4). Continuing to iterate the method after the PSNR peak, the resulting images seem to improve very little visually. This can be explained by the fact that the image has already recovered boundaries and textures, but the noise of the image gets also amplified. To better understand this phenomenon, the result of 2K iterations is shown in Fig. 4 e.

In Fig. 4 f, we use several iterations of the proposed method with total variation regularization (15) to reverse the same noisy filter. Visual inspection and numerical evaluation indicate that the total variation acts as a powerful regularization to the proposed method, which can significantly remove the noise and lead to a much higher PSNR value.

To verify the generality of our approach, we try it with different kinds of blur and noise. We used Gaussian and Poisson noise, respectively, as they are the most common noise one deals with. Experimental results show that the proposed method can significantly remove image blurring (Gaussian blur, Motion blur, or even blur described by an unknown kernel) under these two types of noise. We compare our results with the state-of-the-art deblurring methods [1, 4, 16] in Fig. 5. The performance of the proposed method is between non-blind and blind methods. This observation is not surprising, as compared with blind methods, we can query the filter as a black-box. However, compared to non-blind methods, we do not have exact information about the kernel. From this point of view, the method developed in this work can be treated as a semi-blind deblurring method.

4.3 Image super-resolution

Tao et al. [30] demonstrated that reverse filtering could be used to improve results obtained by image super-resolution methods. In this section, we demonstrate that (14) is also capable to improve image super-resolution results and performs better than [30].

We consider a low-resolution image as a down-sampled (filtered) version of a high-resolution image. Tao et al. proposed in [30] that the down-sampling operation and image super-resolution process could be regarded as a black-box filter, so as to ensure the image size consistency. Let us take bicubic interpolation as an example, one can consider for the black-box filter imresize(imresize(x,1/2,’ bicubic’),2,’ bicubic’) using MATLAB notation. In Fig. 6 a-c, we demonstrate this idea by combining iterations of (13), i.e., Algorithm 1, with bicubic interpolation. The PSNR value shows that the proposed method further improves the quality of the restored image compared to the approach of Tao et al. [30]. However, we observe that the interpolation-based method may produce ringing artifacts in the position where the gradient changes significantly. We note that edge artifacts increase the image gradient. Therefore, it is expected that iterations of (15), with the total variation prior, could help removing this artifact. We verified this fact experimentally in Fig. 6d. With the regularization from the total variation term, the artifacts are removed, and the quality of the restored image has been significantly improved qualitatively and quantitatively.

Fig. 6
figure 6

Image super-resolution enhanced by iterations of (13) and (15)

Full size image

For a more rigorous evaluation, we combine the approach developed in this work with several commonly used image interpolation methods. The best results (selected within 3K iterations for interpolation-based approaches and 300 for SRCNN [5]) are compared to the method of Tao et al. [30] in Table 2. PSNR is used as a quantitative measure for the evaluation. The restored images are shown in Fig. 7 for visual comparison. These results show that the proposed method can enhance various interpolation-based methods, and that it performs better than that of Tao et al. For interpolation-based methods, Tao’s method produces edge artifacts at the edges of the image, such as the junction of the beak with the background. In contrast, this artifact is suppressed in the results obtained via the method proposed in this work, see Fig. 7e and f. Moreover, the proposed method also shows some improvements when the advanced SRCNN approach is used, but the improvements are more nuanced than when interpolation-based methods are used.

Table 2 Comparison of the best results obtained using the proposed method with those of Tao et al. [30]
Full size table
Fig. 7
figure 7

Image super-resolution enhanced by the proposed method and the method of Tao et al. [30]. The number of iterations for Tao’s method is fixed at 50 (a larger number of iterations does not improve the result). The number of iterations for the proposed method is within 3K for interpolation-based methods and 300 for SRCNN [5] method

Full size image

5 Accelerated Landweber-like iterations

Although the proposed method (13) demonstrates good reverse filtering results, it requires many iterations. Below we consider two acceleration schemes for (13).

Acceleration by phase correction. The van Cittert and Landweber methods are the simplest and most fundamental iterative methods for linear inverse problems [3]. One can easily observe that if image filter \({{\varvec{f}}}\,(\cdot )\) is linear, then the van Cittert method corresponds to (4) and (3), while (11) constitutes Landweber iterations in the frequency domain. As noted in [3, Chapter 6], the method of Landweber can be applied to any imaging system, while the van Cittert method requires an imaging system to have some particular properties. In our experiments, we have already seen that (4) and (3) failed to deal with non-symmetric blurring filters. On the other hand, if the van Cittert method converges, it does it faster than the Landweber one, as each iteration of the latter involves reblurring.

For the reverse filtering problem, the advantages of both the van Cittert and Landweber methods can be combined together, if we consider the methods in the frequency domain and, for each iteration, apply a simple phase correction step. Namely, replacing \(H_k^*\) in (13) by \(H_k^*/|H_k|\) yields

$$\begin{aligned} X_{k+1}=X_k-\tau \,\frac{H_k^*}{|H_k|}(F({{\varvec{x}}}_k)-Y). \end{aligned}$$
(17)

As \(H_k^*/|H_k|\) does only a phase correction, it seems natural to call (17) by phase-corrected Landweber iterations.

Figure 8 demonstrates that (17) accelerates (13) and is capable to deliver better results for reversing linear and nonlinear filter (the MT and AMF filters are used).

Setting \(\tau =1\) in (17) leads to a simplified version of (17) which can be considered as a phase-corrected version of the method of Tao et al. [30], as it transfers (3) into the frequency domain and corrects it by a pure phase filter \(H_k^*/|H_k|\). This phase-corrected version of (17) is used to produce Fig. 1 (f) result.

Nesterov-accelerated Landweber-like iterations. Another way to accelerate our basic method (13) is to combine it with the Nesterov gradient descent acceleration scheme [21].

$$\begin{aligned}&H_k=F({{\varvec{z}}}_k)/Z_k,\quad X_{k+1}=Z_{k}-\tau \,H_k^*\left( F({{\varvec{z}}}_k)-Y\right) , \\&Z_{k+1}=X_{k+1}+\frac{k}{k+3} (X_{k+1}-X_k ). \end{aligned}$$

Figure  8 a and g shows that the Nesterov-accelerated iterations converge faster than (13) for linear filters. However, some instability may be developed if reverse filtering with nonlinear filters is considered.

Fig. 8
figure 8

Results obtained with two accelerated schemes (17) and Nesterov-accelerated iterations; \(\tau =1,\,N=1K\)

Full size image

6 Conclusion

In this work, we have considered the reverse image filtering problem and proposed to use Landweber-like iterations for image defiltering purposes. The proposed method is robust to noise and produces state-of-the-art defiltering results for a number of popular linear and nonlinear image filters. In addition to image defiltering, we have verified the effectiveness of our method for image deblurring and super-resolution. For the image super-resolution problem, the proposed method can be applied as a post-processing step and, according to our experiments, is capable of achieving a significant improvement of state-of-the-art image super-resolution methods.

The main limitation of the proposed method is that it demonstrates a slow convergence and too many iterations are required to achieve good defiltering results. We have addressed this limitation by considering and evaluating two acceleration schemes. One of the schemes combines Landweber-like iterations with phase correction steps and, in addition to producing good reverse image filtering results, seems useful for other image processing tasks.