Triangularized Orthogonalization-Free Method for Solving Extreme Eigenvalue Problems

Abstract

A novel orthogonalization-free method together with two specific algorithms is proposed to address extreme eigenvalue problems. On top of gradient-based algorithms, the proposed algorithms modify the multicolumn gradient such that earlier columns are decoupled from later ones. Locally, both algorithms converge linearly with convergence rates depending on eigengaps. Momentum acceleration, exact linesearch, and column locking are incorporated to accelerate algorithms and reduce their computational costs. We demonstrate the efficiency of both algorithms on random matrices with different spectrum distributions and matrices from computational chemistry.

A Proof of Theorem 4

Proof of Theorem 4

All fixed points of (16) satisfy $g_2(X) = 0$. We first analyze the fixed points for a single column case and then complete the proof by induction. Notations used in this proof are the same as those in the proof of Theorem 3.

We denote the single column X as x. Obviously, when $x = 0$, we have $g_2(x) = 0$. Now, consider the nontrivial case $x \ne 0$. The equality $g_2(x) = 0$ can be expanded as,

$$\begin{aligned} \left( \left( 2 - x^\top x\right) A - x^\top A x I \right) x = 0. \end{aligned}$$

(45)

According to (45), for nonzero x, the matrix $B = \left( 2 - x^\top x\right) A - x^\top A x I$ must has a zero eigenvalue and x lies in its corresponding eigenspace. When $x^\top x = 2$, the matrix $B = x^\top A x I$ does not have zero eigenvalue due to the negativity assumption on A. Hence x is parallel to one of A’s eigenvector, i.e., $Ax = \lambda x$. Substituting this into (45), we obtain,

$$\begin{aligned} 2 (1 - x^\top x) \lambda x = 0. \end{aligned}$$

(46)

Since $\lambda < 0$ and $x \ne 0$, we have $x^\top x = 1$. Hence we conclude that for $g_2(x) = 0$, x is either a zero vector or an eigenvector of A.

Now we consider multicolumn case. The first column of $g_2(X) = 0$ is the same as (45). Hence $X_1 = U P_1 S_1$.

Assume the first i columns of X obey $X_i = U P_i S_i$. Then the $(i+1)$-th column of $g_2(X) = 0$ is

$$\begin{aligned} 2A x_{i+1} - Ax_{i+1} x_{i+1}^\top x_{i+1} - x_{i+1} x_{i+1}^\top A x_{i+1} - AX_i X_i^\top x_{i+1} - X_i X_i^\top Ax_{i+1} = 0. \end{aligned}$$

(47)

Obviously, if $x_{i+1} = 0$, then (47) holds. When $x_{i+1} \ne 0$, we left multiply (47) with $X_i^\top$, adopt the commuting property of diagonal matrices, and obtain,

$$\begin{aligned} \begin{aligned} 0 =&S_i P_i^\top \left( 2\Lambda - x_{i+1}^\top x_{i+1} \Lambda - x_{i+1}^\top A x_{i+1} I - \Lambda P_i P_i^\top - \Lambda \right) U^\top x_{i+1} \\ =&- S_i P_i^\top \left( x_{i+1}^\top x_{i+1} \Lambda + x_{i+1}^\top A x_{i+1} I \right) U^\top x_{i+1} \\ \end{aligned} \end{aligned}$$

(48)

where the second equality adopts the fact that $P_i^\top \Lambda P_i P_i^\top = P_i^\top \Lambda$. Due to the negativity of A, we notice that $x_{i+1}^\top x_{i+1} \Lambda + x_{i+1}^\top A x_{i+1} I$ is a diagonal matrix with strictly negative diagonal entries. Hence the equality (48) is equivalent to

$$\begin{aligned} S_i P_i^\top U^\top x_{i+1} = 0. \end{aligned}$$

(49)

As long as (49) holds, we have $X_i^\top x_{i+1} = 0$ and $X_i^\top A x_{i+1} = 0$. Therefore, solving (47) can be addressed via solving

$$\begin{aligned} 2A x_{i+1} - Ax_{i+1} x_{i+1}^\top x_{i+1} - x_{i+1} x_{i+1}^\top A x_{i+1} = 0. \end{aligned}$$

(50)

Hence $x_{i+1}$ satisfies (49). Combining the solution of the single column case (45) and the constraint (49), we conclude that $X_{i+1}$ is of the form $U P_{i+1} S_{i+1}$.

The stabilities of fixed points should also be analyzed through the spectrum properties of their Jacobian matrices. The Jacobian matrix $\,\mathrm {D}g_2(X)$, again, can be written as a p-by-p block matrix. And using the similar argument as in the proof of Theorem 3, $\,\mathrm {D}g_2(X) = \,\mathrm {D}G$ is a block upper triangular matrix whose spectrum is determined by the spectrum of its diagonal blocks. Through a multivariable calculus, we obtain the expression for $J_{ii}$ as,

$$\begin{aligned} J_{ii} = 2A - A X_i X_i^\top - X_i X_i^\top A - A x_i x_i^\top - x_i^\top x_i A - x_i^\top A x_i I - x_i x_i^\top A. \end{aligned}$$

(51)

We first show the stability of the fixed points of form $X = U_p D$. Substituting these points into (51), we have,

$$\begin{aligned} J_{ii} = A - 2 U_i \Lambda _i U_i^\top - 2 \lambda _i u_i u_i^\top - \lambda _i I. \end{aligned}$$

(52)

Since $\lambda _i$ is smaller than all eigenvalues of $A - U_i \Lambda _i U_i^\top$, $A - U_i \Lambda _i U_i^\top - \lambda _i I$ is strictly positive definite. The rest part of (51) is, obviously, positive definite. Hence $J_{ii}$ is strictly positive definite for all $i = 1, 2, \dots , p$ and fixed points of the form $X = U_p D$ are stable fixed points.

Next we show the rest fixed points are not stable. For a fixed point X, we denote the first index s such that $x_s^\top u_s = 0$. Then we estimate $u_s^\top J_{ss} u_s$ as,

$$\begin{aligned} u_s^\top J_{ii} u_s = 2\lambda _s - x_s^\top x_s \lambda _s - x_s^\top A x_s < 0, \end{aligned}$$

(53)

since $x_s^\top x_s \le 1$ and A is negative definite. Therefore, the rest fixed points are not stable. $\square$