Convergence Theory for Preconditioned Eigenvalue Solvers in a Nutshell

Abstract

Preconditioned iterative methods for numerical solution of large matrix eigenvalue problems are increasingly gaining importance in various application areas, ranging from material sciences to data mining. Some of them, e.g., those using multilevel preconditioning for elliptic differential operators or graph Laplacian eigenvalue problems, exhibit almost optimal complexity in practice; i.e., their computational costs to calculate a fixed number of eigenvalues and eigenvectors grow linearly with the matrix problem size. Theoretical justification of their optimality requires convergence rate bounds that do not deteriorate with the increase of the problem size. Such bounds were pioneered by E. D’yakonov over three decades ago, but to date only a handful have been derived, mostly for symmetric eigenvalue problems. Just a few of known bounds are sharp. One of them is proved in doi:10.1016/S0024-3795(01)00461-X for the simplest preconditioned eigensolver with a fixed step size. The original proof has been greatly simplified and shortened in doi:10.1137/080727567 by using a gradient flow integration approach. In the present work, we give an even more succinct proof, using novel ideas based on Karush–Kuhn–Tucker theory and nonlinear programming.

Appendix

I. An alternative estimate for the left-hand side of (4.8), which is sharp with respect to all variables, can be derived as follows: With $\delta :=\beta /\alpha$ and $\varepsilon :=\mu _l/\mu _k$, a new representation of $\gamma ^2$ is given by

$$\begin{aligned} \gamma ^2 =\frac{(\delta -\varepsilon )^2(1+\alpha ^2)}{(1-\varepsilon )^2(1+\alpha ^2\delta ^2)}. \end{aligned}$$

This results in a quadratic equation for $\delta$ with the roots

$$\begin{aligned} \delta _{\pm }=\frac{\varepsilon (1+\alpha ^2)\pm \gamma (1-\varepsilon ) \sqrt{(1+\alpha ^2)(1+\alpha ^2\varepsilon ^2)-\alpha ^2\gamma ^2(1-\varepsilon )^2}}{(1+\alpha ^2)-\alpha ^2\gamma ^2(1-\varepsilon )^2}. \end{aligned}$$

Since $\delta ^2=\dfrac{\beta ^2}{\alpha ^2}=\dfrac{\mu _k-\mu (y)}{\mu (y)-\mu _l}\; \dfrac{\mu (x)-\mu _l}{\mu _k-\mu (x)}$, a strictly sharp bound for the estimate in (4.8) is given by $\max \{\delta _{+}^2,\delta _{-}^2\}=\delta _{+}^2$. We note that in the limit case $\mu (x)\rightarrow \mu _k$ it holds that $\alpha \rightarrow 0$, and $\delta _{+}^2$ turns into $(\varepsilon +\gamma (1-\varepsilon ))^2$. This coincides with the known bound in (4.8).

II. The bound in (4.8) contains a convex combination of 1 and $\mu _l/\mu _k$. Interestingly, this bound can also be derived by using a convex function as follows: Without loss of generality, we assume that x has a positive $x_k$ coordinate. Then, $\angle (x,x_k)$ is an acute angle. Since $B>0, \angle (Bx,x)$ and $\angle (Bx,x_k)$ are also acute angles. The equality (4.1) together with $\gamma <1$ shows further $\angle (Bx,y)<\angle (Bx,x)<\pi /2$. Since $\angle (Bx,x_k)$ and $\angle (Bx,y)$ are acute angles, the vectors $x_k$ and y are located in a half-plane whose boundary line is orthogonal to Bx. A simple case differentiation shows that $\angle (y,x_k)$ is either equal to $|\angle (Bx,x_k)-\angle (Bx,y)|$ or equal to $\angle (Bx,x_k)+\angle (Bx,y)$. Further, we use the equalities

$$\begin{aligned} \tan ^2\angle (y,x_k)=\frac{\mu _k-\mu (y)}{\mu (y)-\mu _l},\;\; \tan ^2\angle (x,x_k)=\frac{\mu _k-\mu (x)}{\mu (x)-\mu _l},\;\; \frac{\tan ^2\angle (Bx,x_k)}{\tan ^2\angle (x,x_k)}=\frac{\mu _l^2}{\mu _k^2}, \end{aligned}$$

which can be derived in a similar way to Sect. 3. The last equality proves $\angle (Bx,x_k)<\angle (x,x_k)$, since the tangent is an increasing function for acute angles, and $\mu _l<\mu _k$. This leads to $\angle (x,x_k)=\angle (Bx,x_k)+\angle (Bx,x)$, since $x, Bx, x_k$ are all in the same quadrant. In summary, it holds that

$$\begin{aligned} \angle (y,x_k)\le \angle (Bx,x_k)+\angle (Bx,y)<\angle (Bx,x_k)+\angle (Bx,x) =\angle (x,x_k)<\pi /2, \end{aligned}$$

(4.9)

i.e., $\angle (y,x_k)$ is a further acute angle. Using these acute angles, we write (4.8) equivalently as

$$\begin{aligned} \tan \angle (y,x_k)\le \gamma \tan \angle (x,x_k) +(1-\gamma )\tan \angle (Bx,x_k). \end{aligned}$$

(4.10)

In order to prove (4.10), we use (4.1) again, together with $\varphi :=\angle (Bx,x), \vartheta :=\angle (Bx,x_k)$ and the first inequality in (4.9). It holds that

$$\begin{aligned} \tan \angle (y,x_k)\le \tan [\vartheta +\arcsin \big (\gamma \sin (\varphi )\big )]=:f(\gamma ). \end{aligned}$$

Because

$$\begin{aligned} f'(\gamma )= \frac{\big (1+f(\gamma )^2\big )\sin (\varphi )}{\sqrt{1-\big (\gamma \sin (\varphi )\big )^2}}\ge 0 \quad \text{ for }\quad \gamma \in [0,1], \end{aligned}$$

$f(\gamma )$ is a monotonically increasing function in [0, 1]. The numerator of $f'(\gamma )$ is also a monotonically increasing function and its denominator is monotonically decreasing in $\gamma \in [0,1]$. These two functions are positive so that $f'(\gamma )$ is also a monotonically increasing function. Thus, $f(\gamma )$ is a convex function in [0, 1], and

$$\begin{aligned} \tan \angle (y,x_k)\le f(\gamma )\le (1-\gamma )f(0)+\gamma f(1) =(1-\gamma )\tan (\vartheta )+\gamma \tan (\vartheta +\varphi ), \end{aligned}$$

which proves (4.10) and hence (4.8).