A Gauss–Newton iteration for Total Least Squares problems

Abstract

The Total Least Squares solution of an overdetermined, approximate linear equation $Ax \approx b$ minimizes a nonlinear function which characterizes the backward error. We devise a variant of the Gauss–Newton iteration with guaranteed convergence to that solution, under classical well-posedness hypotheses. At each iteration, the proposed method requires the solution of an ordinary least squares problem where the matrix A is modified by a rank-one term. In exact arithmetics, the method is equivalent to an inverse power iteration to compute the smallest singular value of the complete matrix $(A\mid b)$. Geometric and computational properties of the method are analyzed in detail and illustrated by numerical examples.

Appendix A: Proof of Theorem 3.1

Before entering into the details of the proof we rephrase the iteration in Algorithm GN-TLS with optimal step length in a more general setting. Notations adopted hereafter mirror those in the previous sections, with some minor exceptions.

Let $C\in \mathbb {R}^{p\times q}$ be a full column rank matrix, let $\mathscr {S}= \{s \in \mathbb {R}^q : \Vert s\Vert = 1\}$ and $\mathscr {E}= \{y \in \mathbb {R}^p : y = Cs, s\in \mathscr {S}\}$. Therefore, $\mathscr {E}$ is a differentiable manifold of $\mathbb {R}^p$; more precisely, it is an ellipsoid whose (nontrivial) semiaxes are directed as the left singular vectors of C; and the corresponding singular values are the respective lengths.

For any nonzero vector $z\in \mathrm {Range}(C)$ there exists a unique vector $y \in \mathscr {E}$ such that $z = \alpha y$ for some scalar $\alpha > 0$; we say that y is the retraction of z onto $\mathscr {E}$.

For any $f\in \mathscr {E}$ let $\mathscr {T}_f$ be the tangent space of $\mathscr {E}$ in f. Hence, $\mathscr {T}_f$ is an affine $(q-1)$-dimensional subspace of $\mathbb {R}^p$ with $f\in \mathscr {T}_f$. If $f = Cs$ then it is not difficult to verify that $\mathscr {T}_f$ admits the following description:

$$\begin{aligned} \mathscr {T}_f = \{ f + Cw,\ s^Tw = 0\} . \end{aligned}$$

In fact, the map $s\mapsto Cs$ transforms tangent spaces of the unit sphere $\mathscr {S}$ into tangent spaces of $\mathscr {E}$.

Consider the following iteration:

• Choose $f_0 \in \mathscr {E}$

• For $k = 0,1,2,\ldots$

  • Let $z_{k}$ be the minimum norm vector in $\mathscr {T}_{f_k}$

  • Let $f_{k+1}$ be the retraction. of $z_k$ onto $\mathscr {E}$.

Owing to Lemma 3.1 it is not difficult to recognize that the sequence $\{f(x_k)\}$ produced by Algorithm GN-TLS with optimal step length fits into the framework of the foregoing iteration.

Hereafter, we analyze the behavior of the sequence $\{f_k\}\subset \mathscr {E}$ and of the auxiliary sequence $\{s_k\}\subset \mathscr {S}$ defined by the equation $s_k = C^+ f_k$. In particular, we will prove that the sequence $\{s_k\}$ is produced by a certain power method and converges to a point in $\mathscr {S}$ corresponding to the smallest semiaxis of $\mathscr {E}$. To this aim, we need the following preliminary result characterizing the solution of the least squares problem with a linear constraint.

Lemma 5.1

Let A be a full column rank matrix and let v be a nonzero vector. The solution $\bar{x}$ of the constrained least squares problem

$$\begin{aligned} \min _{x\, :\, v^Tx = 0}\Vert Ax - b\Vert \end{aligned}$$

is given by $\bar{x} = P x_{\mathrm {LS}}$ where $x_{\mathrm {LS}} = A^+b$ is the solution of the unconstrained least squares problem and

$$\begin{aligned} P = I - \frac{1}{v^T(A^TA)^{-1}v}(A^TA)^{-1}vv^T \end{aligned}$$

is the oblique projector onto $\langle v\rangle ^\perp$ along $(A^TA)^{-1}v$.

Proof

Simple computations using Lagrange multipliers, see e.g., [5, § 6.2.4], prove that $\bar{x}$ fulfills the linear equation

$$\begin{aligned} \begin{pmatrix} A^TA &{} v \\ v^T &{} 0 \end{pmatrix} \begin{pmatrix} \bar{x} \\ \lambda \end{pmatrix} = \begin{pmatrix} A^Tb \\ 0 \end{pmatrix} , \end{aligned}$$

for some scalar $\lambda$. To solve this equation, consider the block triangular factorization

$$\begin{aligned} \begin{pmatrix} A^TA &{} v \\ v^T &{} 0 \end{pmatrix} = \begin{pmatrix} A^TA &{} 0 \\ v^T &{} 1 \end{pmatrix} \begin{pmatrix} I &{} w \\ 0 &{} -v^Tw \end{pmatrix} \end{aligned}$$

where $w = (A^TA)^{-1}v$. Solving the corresponding block triangular systems we get

$$\begin{aligned} \begin{pmatrix} A^TA &{} 0 \\ v^T &{} 1 \end{pmatrix} \begin{pmatrix} x_{\mathrm {LS}} \\ -v^Tx_{\mathrm {LS}} \end{pmatrix} = \begin{pmatrix} A^Tb \\ 0 \end{pmatrix} , \end{aligned}$$

and

$$\begin{aligned} \begin{pmatrix} I &{} w \\ 0 &{} -v^Tw \end{pmatrix} \begin{pmatrix} \bar{x} \\ \lambda \end{pmatrix} = \begin{pmatrix} x_{\mathrm {LS}} \\ -v^Tx_{\mathrm {LS}} \end{pmatrix} , \end{aligned}$$

with $\lambda = -v^Tx_{\mathrm {LS}}/v^Tw$ and

$$\begin{aligned} \bar{x} = x_{\mathrm {LS}} - \lambda w = x_{\mathrm {LS}} - \frac{v^Tx_{\mathrm {LS}}}{v^T(A^TA)^{-1}v} (A^TA)^{-1}v . \end{aligned}$$

The claim follows by rearranging terms in the last formula. $\square$

Let $s_k\in \mathscr {S}$ and let $f_k = Cs_k$ be the corresponding point on $\mathscr {E}$. The minimum norm vector in $\mathscr {T}_{f_k}$ can be expressed as $z_k = f_k + Cw_k$ where

$$\begin{aligned} w_k = \arg \min _{w \, :\, s_k^Tw = 0}\Vert f_k + Cw \Vert . \end{aligned}$$

A straightforward application of Lemma 5.1 yields the formula

$$\begin{aligned} w_k&= -\left( I - \frac{1}{s_k^T(C^TC)^{-1}s_k}(C^TC)^{-1} s_ks_k^T \right) s_k \\&= \frac{1}{s_k^T(C^TC)^{-1}s_k}(C^TC)^{-1}s_k - s_k . \end{aligned}$$

In fact, the solution of the unconstrained problem $\min _w \Vert f_k + Cw \Vert$ clearly is $w_\mathrm {LS}= -s_k$, and $s_k^Ts_k = 1$. Then, the minimum norm vector in $\mathscr {T}_{f_k}$ admits the expression

$$\begin{aligned} z_k = C(s_k + w_k) = \alpha _k C (C^TC)^{-1} s_k , \end{aligned}$$

where $\alpha _k > 0$ is a normalization constant whose exact formula is not relevant. Since $f_{k+1}$ is the retraction of $z_k$ onto $\mathscr {E}$ and $C^+C = I$, we conclude that $f_{k+1} = Cs_{k+1}$ with

$$\begin{aligned} s_{k+1} = C^+ f_{k+1} = \beta _k (C^TC)^{-1} s_k , \quad \beta _k = 1/\Vert (C^TC)^{-1} s_k\Vert . \end{aligned}$$

(5.1)

Finally,

$$\begin{aligned} f_{k+1} = \beta _k C(C^TC)^{-1}C^+ f_k = \beta _k (C^+)^TC^+ f_k = \beta _k (CC^T)^+ f_k, \end{aligned}$$

as $(C^+)^TC^+ = (CC^T)^+$. This proves the first part of Theorem 3.1.

We are now in position to describe the asymptotic behavior of $\{s_k\}$. As shown in Eq. (5.1), the sequence $\{s_k\}$ corresponds to a power method for the matrix $(C^TC)^{-1}$ with normalization. The spectral decomposition of $(C^TC)^{-1}$ can be readily obtained from the SVD $C = U\varSigma V^T$,

$$\begin{aligned} (C^TC)^{-1} = V\varLambda V^T , \quad \varLambda = \mathrm {diag}(\sigma _1^{-2},\ldots ,\sigma _q^{-2}) . \end{aligned}$$

By hypotheses, the eigenvalue $\sigma _q^{-2}$ is simple and dominant, and the angle between the respective eigenvector $v_q$ and the initial vector $s_0$ is acute. For notational simplicity let $\rho = \sigma _q^2/\sigma _{q-1}^2$. Noting that $(C^TC)^{-1}$ is symmetric and positive definite, classical results on convergence properties of the power method [5, §7.3.1] give us the asymptotic converge nce estimates

$$\begin{aligned} \Vert s_k - v_q \Vert= & {} O(\rho ^{k}) , \quad s_k^T (C^TC)^{-1}s_k = \sigma _q^{-2} + O(\rho ^{2k}) ,\\ \Vert (C^TC)^{-1}s_k\Vert= & {} \sigma _q^{-2} + O(\rho ^{2k}) . \end{aligned}$$

Using again (5.1) we get

$$\begin{aligned} \Vert f_k\Vert ^2 = s_{k}^TC^TCs_{k}&= \frac{s_{k-1}^T(C^TC)^{-1}C^TC(C^TC)^{-1}s_{k-1}}{\Vert (C^TC)^{-1}s_{k-1}\Vert ^2} \\&= \frac{s_{k-1}^T(C^TC)^{-1}s_{k-1}}{\Vert (C^TC)^{-1}s_{k-1}\Vert ^2} = \frac{\sigma _q^{-2} + O(\rho ^{2k})}{\sigma _q^{-4} + O(\rho ^{2k})} = \sigma _q^{2} + O(\rho ^{2k}) \end{aligned}$$

and the proof of Theorem 3.1 is now complete.