GKB-GCV Method for Solving Generic Tikhonov Regularization Problems

Abstract

The W-GCV method is one of the iterative methods used to solve a large scale Vtandard form of the Tikhonov regularization, but it is also necessary for solving the general form of the Tikhonov regularization. This paper proposes a new solver, called GKB-GCV, which is an extension of the W-GCV by using the GSVD. Numerical results are presented to show the usefulness of the GKB-GCV method in large scale ill-posed problems.

I. Introduction

The stable approximate solution for a large scale ill-posed problem of the form:

((1))

is computed, where matrix A ∈ ℝ^m×n, m ≥ n is ill- conditioned. The right-hand vector b ∈ ℝ^m contains the following error:

((2))

wherex _exact ∈ ℝⁿ is the exact solution, and ϵ ∈ ℝ^m is the unknown noise. A matrix of this form sometimes comes from image resolutions. Because matrix ∃ is ill-conditioned, x _LS is dependent on noise. The Tikhonov regularization [11] constructs stable approximations of x _exact by solving the least squares problem of the form:

((3))

where L ∈ ℝ^{p × n} is the regularization matrix, and λ > 0 is the regularization parameter. The standard form of the Tikhonov regularization is when L = I _n , where I _n is the n × n identity matrix. The general form of the Tikhonov regularization is when L ≠ I _n . When the common space between the null spaces of ∃ and L is the zero space, the regularization problem (3) has a unique solution. To obtain a good approximate solution for (3), an appropriate regularization parameter is required. There are many methods for determining the regularization parameter without knowledge of the noise’s norm ∥ϵ∥₂, [1, 5, 6].

Solving equation (3) and all the approach selecting parameters of the above is computable when the GSVD for the pair of matrix (∃, L) has been computed. One problem is that the GSVD is not cost effective when the ∃ or the L is a large scale matrix. For a large scale problem, iterative methods are used, e.g. the LSQR, the CGLS, or some kind of Krylov subspace method. Computing good approximate solutions by using iterative methods, require the parameter λa priori and a suitable stopping criteria. The hybrid method solves this issue through combining the projection method with an inner regularization method. For L = I _n , there are two hybrid methods, called GKB-FP [2] and W-GCV [4]. These methods do not require identifying the norm ∥ϵ∥₂, and contain a projection over the Krylov subspace generated by the Golub-Kahan Bidiagonalization (GKB) method. The difference between these two methods is in the approach in terms of determining the regularization parameter. The GKB-FP uses the FP scheme, whereas the W-GCV uses the weighed GCV.

Lampe et al. [8] and Reichel et al. [10] have proposed approaches for L ≠ I _n by minimizing the regularization problem over the generalized Krylov subspace. These approaches determine the regularization parameter by using the knowledge of the norm ∥ϵ∥₂. Bazán et al. [3] proposed an approach without identifying the norm ∥ϵ∥₂, which is created by the extension of the GKB-FP method.

This paper focuses on the W-GCV method which is a solver for a large scale standard form of the Tikhonov regularization which does not require identifying the norm of the noise. This paper proposes applying an extension of the W-GCV to the general form of the Tikhonov regularization. The approach of the W-GCV is based on the idea of the GKB-FP and the idea of the AT-GCV [9]. The stopping criteria of the W-GCV and the AT-GCV are also compared.

This paper is organized as follows: After the introduction,Section II summarizes the framework of the classical W-GCV method. In Section III, the extensions of the W-GCV to the general form of the Tikhonov regularization, are described briefly. Following this, a new scheme of GKB-GCV is proposed. In section IV, the usefulness of the GKB-GCV for test problems, is illustrated. The conclusions and possible future studies are explored in Section V.

II. The W-GCV method

The W-GCV is one of the algorithms for the standard form of the Tikhonov regularization, which is based on the GKB and weighted GCV. For the standard form, i.e. L = I _n , the SVD of matrix ∃ reduces the solution for equation (3) as follows:

where each U = [u ₁, …, u _m ], V = [υ ₁, …, υ _n ] has left and right singular vectors of ∃, and 6 is the singular value of matrix ∃ diagonal with , and σ₁ ≥ … ≥ σ _n ≥0, and 0 on the nondiagonal. When we apply k < n GKB steps to matrix ∃ with the initial vector b, it results in two matrices Y _k+1 = [y ₁, …, y _k+1] ∈ ℝ^n×k and with orthonormal columns, and a lower bidiagonal matrix as follows.

where e _i denotes the i-th unit vector in ℝ^k+1. Furthermore, columns of W _k are the orthonormal basis for the generalized Krylov subsupace K _k (A ^T A, A ^T b. The standard form of regularization, i.e. L = I _n , over the generated Krylov subspace is as follows:

((4))

Since the columns of W _k are the orthonormal basis for the generated Krylov subsupace, equation (4) is reduced as follows:

((5))

This reduction technique is a good choice for large scale problems, because this approach reduces the size of the least squares problem: (m+p) × n to (2k+1) × k.

The GCV and weighted GCV methods determine the regularization parameter. The GCV determines the regularization parameter by searching for the minimum point of function as follows:

((6))

where $A_{\lambda ,L}^{+}={{({{A}^{T}}A+{{\lambda }^{2}}{{L}^{T}}L)}^{-1}}{{A}^{T}}$. Using the SVD for matrix ∃, equation (6) is written as follows:

((7))

The approach of the GCV to the least squares problem is as follows:

((8))

However, the optimal parameter determined by equation (8) is unsuitable for equation (3). Therefore, the weighted GCV for a reduced system of equation (5) is used instead:

((9))

When ω = 1, the weighted GCV is the same as the standard GCV method. Furthermore, the approximate solution becomes smooth at ω > 1, and less smooth at ω < 1. Similarly, the SVD for matrix B _k reduces equation (9) as follows:

((10))

where ${B_k} = {U_k}{\Sigma _k}V_k^T$ . The SVD for B _k can be computed easily, because the size of B _k is (k+1) × k, and smaller than the size of matrix ∃. The stopping criterion is as follows:

((11))

where $\hat G(k)$ is an approximation for ${G_{A,b,{I_n}}}(\lambda )$ without a weighted parameter, and tol is the stopping tolerance.

The W-GCV method is summarized in Algorithm 1.

Algorithm 1

W-GCV

III. GKB-GCV Method

The purpose of this section is to identify a working extension of the W-GCV that can be used with a general form of the Tikhonov regularization. The extension that is proposed will be referred to as the GKB-GCV. In its general form, i.e. L ≠ I _n , equation (3) is reduced by the GKB as follows:

((12))

In equation (12), the size of the least squares problem is (k+1+p) × k.

The same reduction to PROJ-L when solving the general form of the Tikhonov regularization of Bazán [3] is used as follows:

((13))

where LW _k = Q _k R _k , using the QR factorization. For increasing k, the QR factorization can be updated computing k+1 elements by using the summation and a product of the vectors. This approach can be used without limitation of dimension for L, i.e. for any number of p, unlike the AT-GCV method which is one of the hybrid methods using the same GCV. The next step was to consider change points in the GCV. One problem is that when L ≠ I _n , the SVD for matrix ∃ can not reduce the number of the residual norm and trace into the GCV function to form at equation (7) and (10). This problem was addressed by using the GSVD for the pair of matrix (∃, L), A = USZ ^-1 and L = VCZ ^-1, where U = [u ₁, …, u _m], V = [υ ₁, …, υ _p] are orthogonal, Z is nonsingular matrix , and each S, C have s ₁ ≥ … ≥ s _n ≥ 0 and 0 ≤ c ₁ ≤ … ≤ c _n on its diagonals and 0 on its nondiagonals. GSVD for the pair of matrix (∃, L) reduces the GCV function as follows:

((14))

Using a similar computation of $G_{{B_k},{\beta _1}{e_1},{R_k}}^{(\omega )}(\lambda )$, the equation (14) was reduced as follows by using GSVD(B _k , R _k ):

where $\matrix{ {{B_k} = {U_k}{S_k}Z_k^{ - 1},} & {{R_k} = {V_k}{C_k}Z_k^{ - 1}} \cr }$. However, the determination of weight parameter Z is difficult. The AT-GCV method applies a similar function to $\hat G(k)$ for the GCV function at step k [9]:

This approach does not need to determine weight parameter Z. These two functions have different purposes. The GCV function in W-GCV determines the appropriate regularization parameter for the reduced equation (13), and the GCV function at AT-GCV approximates the appropriate regularization parameter for the original equation (3). Furthermore, the AT-GCV uses the residual norm entered when computing the GCV function for the stopping rule which is different from the W-GCV:

((15))

where $r_{{\lambda _k}}^{(k)} = {B_k}y_{{\lambda _k}}^{(k)} - {\beta _1}{e_1}$. Numerical experiments were used to illustrate the differences between these stopping rules. The stopping rule for equation (11) to the $\hat G$) and the rule of equation (15) to GKB-GCV(r ^(k)) were noted. The stopping rule for GKB-GCV(r ^(k)) was too severe compared to the stopping rule for GKB-GCV($\hat G$). Hence, to create tolerance with regards to the stopping rule, $\sqrt {tol}$ on GKB-GCV(r ^(k)) was used. In addition, another stopping rule was used:

((16))

The GKB-GCV was compactly summarized in Algorithm 2.

Figure 1:

Change of relative error norm for the increase in k for the three test problem, blur, tomo and heat.

Algorithm 2

GKB-GCV

IV. Convergence analysis of the GKB-GCV method

In this section, we provide convergence properties of the GKB-GCV method. We define the appropriate parameters λ_* and λ _k as follows.

where λ _n = λ_*. Using triangle inequality, following inequality is satisfied.

((17))

The first term of equation (17) is corresponding to an error which occurs as a consequence of stabilization. The second term of (17) will converge monotonically for increasing k, and we verify its convergence property by following experiments.

A. Behavior of relative error norm at each GKB-FP iteration

We use built-in data in MATLAB, blur, tomo and heat, fortest problem, and we use n=30 for blur and tomo, and n=900 for heat. In this time, we don’t use stopping criteria and continue iteration until iteration number arrving at matrix size. In the figure.1 , solid lines represent the left-hand side of (17) and dashed lines represent the right-hand side of (17). For blur and tomo, we could bound well, but the right-hand side of (17) is too larger than the left-hand side of (17) for heat. The reason why the right-hand side of (17) is too large is that for the regularizaiton, relartive error norm rise or fall down after arriving at minimum relative error norm.

V. Numerical experiments

The PROJ-L method which is also one of the hybrid methods and an extension of GKB-FP, was used for the purpose of comparision with the proposed method in this paper. The 2D image deblurring problem which is the procedure for recovering original images from blurred images using noise from the form equation (2) was considered. Matrix ∃ was theblurring operator, e.g. the Point Spread Function(PSF) matrix, and b _exact = A x _exact are blurred images without any noise. All computations of numerical experiments were carried out in MATLAB R2013b, and generated noise vectors ∊ by the MATLAB code randn and NL = ∥∊∥₂/∥b _exact∥₂. N × N images with N ² × N ² Gaussian PSF matrices as a blurring operator were used. The Gaussian PSF was defined by A = (2πσ²)^-1 T ⊗ T, where ς was the parameter used to control the width of the Gaussian PSF, and T was an N × N symmetric banded toeplitz matrix with generators of the form:

from Kilmore et al. [7]. In our tests, we used ς = 2 and band = 16. A regularization matrix lowering the gap between adjacent points was chosen:

Table 1: Results for the test problem rice64 with tol = 10^-4

A. Test problem 1: rice64

The interpolation data of MATLAB, rice image were usedin test problem 1. Firstly, a 64×64 sub-image of rice and rice64 were used to compare noise levels. A ∈ ℝ ^4096×4096 and L ∈ ℝ ^8064×4096, and the condition number was cond(A) ≈ 2.14×10¹⁶. These experiments used ten noise vectors for each noise level: NL = 10^-2, 10^-3 and 10^-4. To simplify the notation, $\bar \lambda ,\bar t$ and Ē the average value of the regularization parameter, time and relative error, and k _m(k _M ) denoted the minimum (maximum) number of steps required.

The computation of the FP method on PROJ-L started with p ₀ = 10 and Π= 1. The stopping criteria was set to tol = 10^-4.

All proposed methods converged faster than the existing method PROJ-L from Table 1. Previous experiments suggested to us that PROJ-L converged comparatively faster in all of the solvers for the general form of the Tikhonov regularization, and were dependent on the noise level. The GKB-GCV(r ^(k)) and the GKB-GCV(r ⁽¹⁾) also were dependent on noise level. The GKB-GCV(r ^(k)) had the same dependence as the PROJ-L. The dependence of the GKB-GCV(r ⁽¹⁾) was smaller than that of the GKB-GCV(r ^(k)). The results for the GKB-GCV(r ⁽¹⁾) were not much different for NL = 10^-2 and 10^-3. Regarding numerical precision, all results of the proposed methods were worse than PROJ-L, except for the results of the GKB-GCV(r ⁽¹⁾) for NL = 10^-2. Because the GKB-GCV($\hat G$) is independent of noise levels unlike the other methods, the relative error did not decrease with the noise level. The relative error of the GKB-GCV($\hat G$) was about 1.5 times as more than the PROJ-L when the noise level was small.

GKB-GCV($\hat G$) is independent of noise levels unlike the other methods, the relative error did not decrease with the noise level. The relative error of the GKB-GCV($\hat G$) was about 1.5 times as more than the PROJ-L when the noise level was small.

B. Test problem 2: rice

The next step was to create a 256 × 256 original image of rice. A ∈ ℝ ^65536×65536, L ∈ ℝ ^{130560×65536} and cond(A) ≈ 3.40×10¹⁶. The same experiment was performed using NL = 10^-2 and 10^-3 this time. The same notation was used for the case of rice64. The computation of the FP method on PROJ-L started with p ₀ = 15 and Π= 1, and tol = 10^-4, for the stopping criteria.

Similar results were obtained for rice64, with the exception of GKB-GCV(r ⁽¹⁾) for NL = 10^-2. Results from the GKB-GCV(r ⁽¹⁾) for NL = 10^-2 was slow with less numerical accuracy than the GKB-GCV(r ^(k)) and the PROJ-L. One of the reason for this, is that hybrid methods do not have properties of monotone convergence. The stopping rules ofhybrid methods must not be too severe or too easy. An easy approach for solving this problem was employed, which used two stopping rules. Specifically, both equations (15) and (16) were applied to the stopping rule. Please see Figure 1 for the original image, deblurred image, and resolution images.

VI. Conclusion

The GKB-GCV is a new solver for the general form of the Tikhonov regularization problem; it is based on the W-GCV. The GKB-GCV was compared to the PROJ-L and the GKB-GCV, using different stopping rules by using two image deblurring problems. The results of the numerical experiments showed that each of the proposed methods had good advantages. The GKB-GCV($\hat G$) was the fastest, although its numerical precision was the worst in all scenarios. This was because the GKB-GCV($\hat G$) does not depend on noise level.

Table 2: Results for the test problem rice with tol = 10^-4

Secondly, the GKB-GCV(r ^(k)) was very fast, but had less accuracy compared to the PROJ-L for all noise levels, while it had the the same dependency on noise levels to PROJ-L. Lastly, the GKB-GCV(r ⁽¹⁾) had a smaller dependence on noise level than GKB-GCV(r ^(k)) and PROJ-L.

Figure 2:

Resolution image of rice with NL = 10^-2