Reorthogonalized block classical Gram–Schmidt

Abstract

A reorthogonalized block classical Gram–Schmidt algorithm is proposed that factors a full column rank matrix $A$ into $A=QR$ where $Q$ is left orthogonal (has orthonormal columns) and $R$ is upper triangular and nonsingular. This block Gram–Schmidt algorithm can be implemented using matrix–matrix operations making it more efficient on modern architectures than orthogonal factorization algorithms based upon matrix-vector operations and purely vector operations. Gram–Schmidt orthogonal factorizations are important in the stable implementation of Krylov space methods such as GMRES and in approaches to modifying orthogonal factorizations when columns and rows are added or deleted from a matrix. With appropriate assumptions about the diagonal blocks of $R$, the algorithm, when implemented in floating point arithmetic with machine unit $\varepsilon _M$, produces $Q$ and $R$ such that $\Vert I- Q ^T\!~ Q \Vert =O(\varepsilon _M)$ and $\Vert A-QR \Vert =O(\varepsilon _M\Vert A \Vert )$. The first of these bounds has not been shown for a block Gram–Schmidt procedure before. As consequence of these results, we provide a different analysis, with a slightly different assumption, that re-establishes a bound of Giraud et al. (Num Math, 101(1):87–100, 2005) for the CGS2 algorithm.

Appendix. Proofs of technical lemmas from Sect. 3

Two of the technical lemmas in Sect. 3, Lemmas 3.1 and 3.3 are based upon matrix multiply and add operations, thus those proofs are presented first. The proof of Lemma 3.2 and the proof of Lemma 3.5 are verifications that two norm bounds in exact arithmetic are not significantly altered in floating point arithmetic.

Using a result from Higham [10, p.71] on matrix multiplication in floating point arithmetic and putting in an additional term for a matrix add, the operation

$$\begin{aligned} M= C +GH, \quad \begin{array}{l} M,C \in \mathbb R ^{m \times p} \\ G \in \mathbb R ^{m \times t}, H \in \mathbb R ^{t \times p}, \\ m \ge t \ge p \end{array} \end{aligned}$$

(6.1)

satisfies

$$\begin{aligned} M+ \Delta M = C+GH \end{aligned}$$

(6.2)

where

$$\begin{aligned} |\Delta M| \le d_{mat} \varepsilon _Mt (|C| + |G||H|) \end{aligned}$$

for some modest constant $d_{mat}$. If we use the bound on the two-norm

$$\begin{aligned} \Vert M\Vert \le \Vert |M|\Vert \le \min \{ m^{1/2} , p^{1/2}\} \Vert M\Vert \end{aligned}$$

for an $M \in \mathbb R ^{m \times p}, m \ge p$, then we have

$$\begin{aligned} \Vert \Delta M\Vert \le d_{mat} \varepsilon _Mt p^{1/2} (\Vert C\Vert + t^{1/2} \Vert G\Vert \Vert H\Vert ). \end{aligned}$$

(6.3)

Thus the bound on these operations for Functions 2.1 and 2.2 in Lemmas 3.1 and 3.3 are just a matter of bounding $\Vert C\Vert$, $\Vert G\Vert$ and $\Vert H\Vert$ in the appropriate context.

First, we prove Lemma 3.1 which concerns the operations of the form (6.1) in Function 2.1.

Proof of Lemma 3.1

Interpreting (6.3) for the computation in (3.18), we have

$$\begin{aligned} \Vert \delta \bar{S}\Vert \le d_{mat} \varepsilon _Mm p^{1/2} t^{1/2} \Vert U\Vert \Vert B\Vert . \end{aligned}$$

By the assumption (3.15), $\Vert U\Vert \le c_U(\varepsilon _M) \le \sqrt{2}$, thus

$$\begin{aligned} \Vert \delta \bar{S}\Vert&\le \sqrt{2} d_{mat} \varepsilon _Mm p^{1/2} t^{1/2} \Vert B\Vert \\&= d_2 \varepsilon _Mm p^{1/2} t^{1/2} \Vert B\Vert , \quad d_2 = \sqrt{2}d_{mat}, \\&= \varepsilon _ML_2(m,t,p) \Vert B\Vert \end{aligned}$$

which is the bound in (3.18).

The second such operation is given in (3.19). The interpretation of (6.3) for that computation is

$$\begin{aligned} \Vert \delta Y\Vert \le d_{mat} \varepsilon _Mt p^{1/2} [ \Vert B\Vert + t^{1/2} \Vert U\Vert \Vert \bar{S}\Vert ]. \end{aligned}$$

By our assumptions below (3.13),

$$\begin{aligned} \Vert \bar{S}\Vert&\le \Vert U^T B\Vert + \Vert \delta \bar{S}\Vert \\&\le \Vert U\Vert \Vert B\Vert + \varepsilon _ML_2(m,t,p) \Vert B\Vert \\&\le ( 1 + \varepsilon _M( 0.5 f_1(m,t,p) + L_2(m,t,p))\Vert B\Vert \\&\le c_S(\varepsilon _M) \Vert B\Vert \\&\le \sqrt{2} \Vert B\Vert . \end{aligned}$$

By (3.15), we have that $\Vert U\Vert \le \sqrt{2}$, thus

$$\begin{aligned} \Vert \delta Y\Vert&\le d_{mat} \varepsilon _Mt p^{1/2} (1 + 2t^{1/2}) \Vert B\Vert \\&= d_3 \varepsilon _Mt p^{1/2} (1 + 2t^{1/2}) \Vert B\Vert \\&= \varepsilon _ML_3(t,p) \Vert B\Vert \end{aligned}$$

which is the bound in (3.19). $\square$

Lemma 3.3 concerns two computations in Function 2.2 that are of the form (6.2).

Proof of Lemma 3.3

Interpreting (6.3) for the computation in (3.32) yields

$$\begin{aligned} \Vert \delta S_B\Vert \le d_{mat} \varepsilon _Mp^{3/2} [\Vert \bar{S}_1\Vert + p^{1/2} \Vert \bar{S}_2\Vert \Vert \bar{R}_1\Vert ]. \end{aligned}$$

To bound the norms, we have that

$$\begin{aligned} \Vert \bar{S}_1\Vert&\le c_B(\varepsilon _M) \Vert B\Vert \le \sqrt{2} \Vert B\Vert , \\ \Vert \bar{S}_2\Vert&\le c_B(\varepsilon _M) \Vert \bar{Q}_1\Vert \le c_B(\varepsilon _M)c_Q(\varepsilon _M) \le 2,\\ \Vert \bar{R}_1\Vert&\le c_R(\varepsilon _M)\Vert B\Vert \le \sqrt{2} \Vert B\Vert . \end{aligned}$$

Thus

$$\begin{aligned} \Vert \delta S_B\Vert&\le d_{mat} \varepsilon _Mp^{3/2} [\sqrt{2}+ 2\sqrt{2} p^{1/2}]\Vert B\Vert \\&\le d_4 \varepsilon _Mp^{3/2} [1+ 2 p^{1/2}]\Vert B\Vert , \quad d_4 = \sqrt{2} d_{mat} \\&= \varepsilon _ML_4(p) \Vert B\Vert . \end{aligned}$$

The final such operation is that in (3.33). The interpretation of (6.3) is

$$\begin{aligned} \Vert \delta R_B\Vert \le d_{mat} \varepsilon _Mp^2 \Vert \bar{R}_1\Vert \Vert \bar{R}_2\Vert . \end{aligned}$$

Using the bounds in (3.30)–(3.31), this is

$$\begin{aligned} \Vert \delta R_B\Vert&\le d_{mat} \varepsilon _Mp^2 c_R^2(\varepsilon _M) c_Q(\varepsilon _M) \Vert B\Vert \\&\le 2\sqrt{2} d_{mat}\varepsilon _Mp^2 \Vert B\Vert \\&= d_5 \varepsilon _Mp^2 \Vert B\Vert , \quad d_5 =2\sqrt{2} d_{mat} \\&= \varepsilon _ML_5(p) \Vert B\Vert . \square \end{aligned}$$

Lemma 3.2, our next technical lemma, is a bound on $\Vert I_m-UU^T\Vert$ where $U\in \mathbb R ^{m \times t}$ is near left orthogonal.

Proof of Lemma 3.2

Let $U$ have the Q-R decomposition

$$\begin{aligned} U=Z \left(\begin{array}{c} R_U \\ 0_{(m-t)\times t }\end{array}\right) \end{aligned}$$

where $Z$ is orthogonal and $R_U$ is upper triangular. Then

$$\begin{aligned} \Vert I_t-U^T\!~U\Vert = \Vert I_t-R_U^T\!~R_U\Vert \le \varepsilon _Mf_1(m,t,p) \end{aligned}$$

and

$$\begin{aligned} I_m-UU^T = Z \left(\begin{array}{cc} I_t -R_UR_U^T&0 \\ 0&I_{m-t} \end{array} \right) Z^T. \end{aligned}$$

Since $R_U^T\!~R_U$ and $R_UR_U^T$ have the same eigenvalues,

$$\begin{aligned} \Vert I_t-R_UR_U^T\Vert =\Vert I_t -R_U^T\!~R_U\Vert \le \varepsilon _Mf_1(m,t,p). \end{aligned}$$

Using the assumption (3.15) and the results of Theorem 3.1, we have

$$\begin{aligned} \Vert I_m-UU^T\Vert&\le \max \{ \Vert I_t-R_UR_U^T\Vert ,1\}\nonumber \\&= \max \{ \Vert I_t-R_U^T\!~R_U\Vert ,1\}\nonumber \\&\le \max \{ \varepsilon _Mf_1(m,t,p),1\} = 1 \end{aligned}$$

(6.4)

by our assumption about $f_1(m,t,p)$ in (1.4). $\square$

Lemma 3.5, our final technical lemma, is a result relating $\Vert \bar{R}_2^{-1}\Vert$ to $\Vert U^T \bar{Q}_1 \bar{R}_2^{-1}\Vert$. It has the elementary but long proof given next.

Proof of Lemma 3.5

We start with interpreting Lemma 3.1 for the second CGS step in Function 2.2 which leads to

$$\begin{aligned} Q_B \bar{R}_2 = (I_m-UU^T)\bar{Q}_1 + F_2. \end{aligned}$$

Taking the normal equations matrices of both sides yields

$$\begin{aligned} \bar{R}_2^T Q_B^T\!~Q_B \bar{R}_2&= \bar{Q}_1^T (I_m-UU^T)^2 \bar{Q}_1 + F_2^T(I_m-UU^T)\bar{Q}_1\nonumber \\&+\bar{Q}_1^T (I_m-UU^T) F_2 + F_2^T\!~F_2. \end{aligned}$$

(6.5)

An expansion of $\bar{Q}_1^T (I_m-UU^T)^2 \bar{Q}_1$ produces

$$\begin{aligned} \bar{Q}_1^T (I_m-UU^T)^2 \bar{Q}_1&= \bar{Q}_1^T\!~\bar{Q}_1-\bar{Q}_1^T UU^T \bar{Q}_1-\bar{Q}_1^T U(I_t-U^T\!~U)U^T \bar{Q}_1 \nonumber \\&= I_p-\bar{Q}_1^T UU^T \bar{Q}_1 +\bar{Q}_1^T\!~\bar{Q}_1-I_p\nonumber \\&-\bar{Q}_1^T U(I_t-U^T\!~U)U^T \bar{Q}_1 \end{aligned}$$

(6.6)

so that the combination of (6.5) and (6.6) is

$$\begin{aligned} \bar{R}_2^T\!~\bar{R}_2 = I_p-\bar{Q}_1^T UU^T \bar{Q}_1 + E \end{aligned}$$

(6.7)

where

$$\begin{aligned} E&= E_1+E_2+E_3, \\ E_1&= F_2^T(I_m-UU^T)\bar{Q}_1+\bar{Q}_1^T (I_m-UU^T) F_2 + F_2^T\!~F_2,\\ E_2&= \bar{Q}_1^T\!~\bar{Q}_1-I_p-\bar{Q}_1^T U(I_t-U^T\!~U)U^T \bar{Q}_1, \\ E_3&= \bar{R}_2^T (Q_B^T\!~Q_B-I_p) \bar{R}_2. \end{aligned}$$

Standard norm bounds for $E_1$ and $E_3$ yields

$$\begin{aligned} \Vert E_1\Vert&\le 2\Vert F_2\Vert \Vert \bar{Q}_1\Vert \Vert I_m-UU^T\Vert + \Vert F_2\Vert ^2 \\&\le 2 \Vert F_2\Vert ( \Vert \bar{Q}_1\Vert +0.5 \Vert F_2\Vert ) \\&\le 2 \varepsilon _Mc_Q(\varepsilon _M) L_F(m,t,p) (c_Q(\varepsilon _M)+ 0.5 \varepsilon _Mc_Q(\varepsilon _M) L_F(m,t,p)) \\&= 2 c_Q^2(\varepsilon _M) L_F(m,t,p)(1+0.5 L_F(m,t,p)) \\&\le 2c_Q^2(\varepsilon _M) c_F(\varepsilon _M) L_F(m,t,p) \\&\le 4\sqrt{2}L_F(m,t,p) \\ \Vert E_3\Vert&= \Vert I_p-Q_B^T\!~Q_B\Vert \Vert \bar{R}_2\Vert ^2 \le \varepsilon _Mc_R^2(\varepsilon _M) c_Q^2(\varepsilon _M) L_1(m,p) \le 4\varepsilon _ML_1(m,p) \end{aligned}$$

whereas for $E_2$ if we use the fact that

$$\begin{aligned} \Vert U^T \bar{Q}_1\Vert&\le \Vert U^T \bar{Q}_1 \bar{R}_2^{-1}\Vert \Vert \bar{R}_2\Vert \nonumber \\&\le c_R(\varepsilon _M) c_Q(\varepsilon _M) \gamma (k), \end{aligned}$$

(6.8)

then

$$\begin{aligned} \Vert E_2\Vert&\le \Vert I_p-\bar{Q}_1^T\!~\bar{Q}_1\Vert +\Vert I_t-U^T\!~U\Vert \Vert U^T \bar{Q}_1\Vert ^2 \\ U&\le \varepsilon _M[L_1(m,p)+ f_1(m,t,p) \gamma ^2(k) c_R^2( \varepsilon _M) c_Q^2(\varepsilon _M)]\\&\le \varepsilon _M[L_1(m,p)+ \gamma (k)c_R^2(\varepsilon _M) c_Q^2(\varepsilon _M) L_F(m,t,p) ] ,\\&\le \varepsilon _M[L_1(m,p) + 4 L_F(m,t,p) ] \end{aligned}$$

since $\gamma (k) \le 1$, $c_R(\varepsilon _M) \le \sqrt{2}$, and $c_Q(\varepsilon _M) \le \sqrt{2}$. Thus, using (3.4), we have

$$\begin{aligned} \Vert E\Vert&\le \Vert E_1\Vert +\Vert E_2\Vert + \Vert E_3\Vert \\&\le \varepsilon _M[4(1+\sqrt{2}) L_F(m,t,p)+5L_1(m,p)] \\&= \varepsilon _ML_6(m,t,p). \end{aligned}$$

Now to show the equivalence between (3.45) and (3.46). Since we assume that $\bar{R}_2$ is nonsingular, we can rewrite (6.7) as

$$\begin{aligned} I_p = \bar{R}_2^{-T} \bar{R}_2^{-1} - \bar{R}_2^{-T}\bar{Q}_1^T U U^T \bar{Q}_1 \bar{R}_2^{-1}+ \bar{R}_2^{-T}E \bar{R}_2^{-1} \end{aligned}$$

so that

$$\begin{aligned} \bar{R}_2^{-T} \bar{R}_2^{-1}=I_p+ \bar{R}_2^{-T}\bar{Q}_1^T U U^T \bar{Q}_1 \bar{R}_2^{-1}- \bar{R}_2^{-T}E \bar{R}_2^{-1}. \end{aligned}$$

(6.9)

If $\lambda _{max}(\cdot )$ is the maximum absolute eigenvalue of the contents, then (6.9) plus Weyl’s inequality for eigenvalues [8, Corollary 8.1.6] and a norm bound yield

$$\begin{aligned} \lambda _{max}( \bar{R}_2^{-T} \bar{R}_2^{-1})=1+ \lambda _{max}(\bar{R}_2^{-T}\bar{Q}_1 U U^T \bar{Q}_1 \bar{R}_2^{-1})- \xi \Vert \bar{R}_2^{-1}\Vert ^2 \end{aligned}$$

(6.10)

where

$$\begin{aligned} |\xi | \le \Vert E\Vert . \end{aligned}$$

(6.11)

Using the relationship, $\lambda _{max}(C^T\!~C)=\Vert C\Vert ^2$ on (6.10) yields (3.44). $\square$