1 Introduction

In this work we investigate modifications of linear fixed-point iterations for computing approximate solutions of a linear equation

$$ Au=f $$
(1.1)

in a Banach space V. A standard linear iterative method for solving (1.1) takes the form

$$ u_{m+1}=Mu_{m}+Nf $$
(1.2)

and corresponds to a linear fixed-point equation

$$ u=Mu+Nf. $$
(1.3)

The matrix M is called the iteration matrix of the method. When (1.3) is related to (1.1) via M = I − NA, then N or its inverse is called the preconditioner of A. If the choice of N is a concrete function of A, then this function defines a class of iterative solvers for all (invertible) A. For instance, for the Jacobi method N is the inverse of the diagonal part of A. In other cases it may be a polynomial of A, and so on. However, in this work we will make direct assumptions on the properties of M and Nf, which may or may not be realizable for a given A. Hence we take the fixed-point problem (1.3) as the starting point of our considerations.

In a practical implementation, if the dimension of V is very large or (in principle) infinite, the use of data-sparse representations of its elements is required for storing the iterates and performing the matrix-vector products. An example that motivates our work is low-rank matrix and tensor formats which can be used for the numerical treatment of high-dimensional problems, and have found many applications in scientific computing [5, 13, 16, 17, 19]. In these formats the numerical complexity for storing vectors and performing basic linear algebra operations is typically captured by one or several rank functions, \(\text {rank}\colon \mathbf {V}\rightarrow \mathbb {N}\cup \{+\infty \}\). These rank functions are usually sub-additive

$$ \text{rank}(u+v)\leq\text{ rank}(u)+\text{ rank}(v), $$

and satisfy rank(0) = 0. More generally, such a sub-additive rank function may arise from a generating set \(\mathcal {D} \subseteq \mathbf {V}\), typically called a dictionary, by defining rank(u) for u≠ 0 as the minimal number of elements from \(\mathcal {D}\) needed to write u as a linear combination, or \(+\infty \) if u is not in the finite span. The goal is then to find possibly sparse, that is low-rank representations of a sought solution in the dictionary. This very general concept of expansion in dictionaries occurs frequently in nonlinear approximation, and covers classical sparsity (then the dictionary consists of unit vectors) or more general best M-term approximation problems. For low-rank matrices or tensors the dictionary consists of all elementary tensors. Of course, when using data-sparse representations with respect to a dictionary, it is implicitly assumed that the true solution of the problem admits accurate ‘low-rank’ approximations, but verifying this analytically in advance can be difficult depending on the application. Also note that in many applications the choice of the dictionary is not only motivated by reducing the numerical complexity, but has some well defined and problem dependent purpose for revealing the structure, patterns, principal subspaces etc. of some measured data.

In this paper we investigate the rank accumulation in iterative methods like the linear fixed-point iteration (1.2) in relation to its convergence speed. When looking at (1.2) we see that a rank increase occurs due to two operations: the application of the operator M and the addition of Nf. To deal with the first we assume a multiplicative model, in which the rank of Mum (and typically also the cost of forming Mum) is proportional to the rank of um, that is, rank(Mum) ≤ μ1rank(um). Then

$$ \text{rank}(u_{m+1}) \le \mu_{1} \text{ rank}(u_{m}) + \text{ rank}(Nf). $$
(1.4)

For several steps one can either turn this into an exponentially growing bound, or draw upon refined estimates on how powers Mℓ or polynomials p(M) increase the rank.

Here we can mention two examples. In sparse approximation in \(\mathbf {V} = \mathbb {R}^{n}\), when the rank of a vector is defined as the number of its nonzero elements, a banded matrix M can be efficiently applied to a sparse vector, but will increase the number of nonzero entries by a multiple of the bandwidth. The bandwidth of Mℓ, however, does not grow exponentially but only linearly in ℓ. In fact, for the same reason the bandwith of any polynomial pℓ(M) of degree at most ℓ grows only linearly. Note that if in the initial linear equation (1.1) the matrix A has a banded structure and N = D is a diagonal preconditioner (e.g. in the Jacobi method), then M = I − DA has the same band structure.

As a second example assume that (1.3) is a fixed point matrix equation in \(\mathbf {V} = \mathbb {R}^{n \times n}\) and M is a Kronecker product operator, \(M = \tilde {M}_{1} \otimes \tilde {M}_{2}\). It is well known that a matrix Sylvester equation, that is, (1.1) with \(\mathbf {A} = \tilde {A}_{1} \otimes I + I \otimes \tilde {A}_{2}\), can be transformed into such a problem with M having spectral radius less than one, if both \(\tilde {A}_{1}\) and \(\tilde {A}_{2}\) have eigenvalues with negative real part [25]. The Kronecker product operator M can then be efficiently applied to a low-rank matrix (in the usual sense) and does not increase the rank at all. Therefore, applying a polynomial pℓ(M) of degree ℓ to a matrix increases the rank at most by a factor of ℓ + 1; see Section 3.1 for a more general example.

Our main attention in this work is on the second step in the iteration (1.2) which is the addition of Nf. In standard applications, Nf is a fully populated vector and the inequality (1.4) indicates that even one such step is infeasible if rank(Nf) is very large or infinite. The standard approach would be replacing Nf by an approximation \(N \tilde f\) of acceptable rank, and will be discussed in Section 3.2. As an alternative, we propose using a sequence gm → Nf of approximations with (usually) growing rank. This leads to the modified fixed-point iteration

$$ \hat{u}_{m+1}=M \hat u_{m}+ g_{m+1}, $$
(1.5)

considered in this paper. It turns out that the \(\hat u_{m}\) converge to a solution u of (1.3) at a similar speed as the standard iteration if the convergence gm → Nf is fast enough.

The proposed modification (1.5) of the fixed-point iteration is an interesting and easily realizable variation of the standard iteration. While several questions could be considered, we focus on its impact on the rank accumulation in certain model cases for M and Nf to show that it can be beneficial compared to the standard iteration. Specifically, to limit the scope we investigate only the cases that the approximation gm → Nf converges exponentially fast, and that the rank amplification by the repeated application of the iteration matrix is either exponential or linear in the number of steps. The latter scenario is motivated by the examples mentioned above.

Besides its implications on the computational cost in iterative methods, our rank estimates also serve a theoretic purpose of characterizing low-rank approximability of structured linear equations since they yield upper bounds on the corresponding approximation numbers

$$ \tau_{r}(u) = \underset{\text{ rank}(v) \le r}{\inf} \frac{\| u - v\| }{\| u \|} $$
(1.6)

for the solution u in terms of the rank parameter r. However, the asymptotic rates obtained in this way by using linear fixed-point iterations (or modifications of them) are not necessarily optimal and can be slow, and hence mainly relevant if V is of very large or infinite dimension. For the two mentioned examples of sparse approximation of systems with banded matrices, and matrix or tensor equations with Sylvester-type operators, better rates than those implied by our analysis (covered by Examples 3.3 and 3.7) are available for Hilbert spaces based on different and more problem related approximation schemes; see, e.g., [4, 12] and [10, 14, 15, 20, 22, 26], respectively.

Furthermore, by taking the fixed-point formulation (1.3) as the point of departure we avoid any discussion under which conditions it is actually possible to design for a given linear equation (1.1) an iterative solver (1.2) for which M is highly contractive and mildly rank increasing at the same time (which could be conflicting targets), while Nf admits fast converging and available low-rank approximations. The existence of such a linear iteration would directly imply that the solution can be well approximated in the dictionary. Clearly, if it is not available, it can be more efficient to use few steps of a method with a general spectrally equivalent preconditioner N and then apply truncation. This difficult question is at the very core of understanding low-rank approximability and preconditioning of linear systems for a given rank function, and should not be treated in a general setup like in this paper. There is, however, an interesting converse logic to this. If the solution u of a given linear equation (1.1) does not satisfy the approximability estimates that would be implied by certain assumptions on M and N, then it means that there cannot exist a linear iterative solver with the desired properties. While this may sound trivial, it can actually be seen as a remarkable non-existence result, say for matrix decompositions A = N− 1(I − M).

Our results extend the studies in [21], where mainly the Richardson iteration and the steepest descent method have been considered, to the broad class of linear iterations (1.2). Of course, a great amount of other iterative methods for low-rank solutions of linear systems has been developed in the literature, among them those based on variational formulations, nonlinear optimization, or greedy methods. One common feature of such methods, that should also be applied in a practical implementation of the modified iterations considered in this paper, is the rank truncation of intermediate iterates. A truncation usually occurs either as an adaptive projection (hard thresholding) or as a prox-operator (soft thresholding), and its combination with fixed-point iterations can be studied in a quite general context; see, e.g. [3, 7, 8, 11] for sparse vectors and [1, 2, 6, 18, 23, 24] for low-rank tensors. In particular, if a certain low-rank approximability of the solution is already known or assumed, then suitable adaptive truncation schemes can lead to refined and near-optimal error estimates. This is, however, outside the scope of the present paper.

The paper is outlined as follows. In Section 2 the convergence rate of the modified iteration (1.5) is estimated for the model case that the gm approximate Nf exponentially fast. In Sections 3.1–3.4 rank estimates for approximate solutions obtained from the standard and modified iteration are derived from assumptions on the rank increasing properties of the iteration matrix M. Section 3.5 presents numerical comparisons of the obtained bounds.

2 Convergence of the Modified Iteration

Let u be a solution to the linear fixed-point equation (1.3). Given 0 < ε ≤ 1 we seek an approximation uε to u of relative accuracy ε, that is,

$$ \frac{\|u-u_{\varepsilon}\|}{\|u\|}\leq\varepsilon. $$
(2.1)

Such a uε will be called an ε-solution of the fixed-point equation (1.3). Here the choice of the norm (and hence of the Banach space V) is usually problem dependent and can already account, e.g., for a large condition number or unboundedness of the operator A in the initial linear equation (1.1). In particular, we assume that M satisfies

$$ \|M\| \leq\zeta<1, $$
(2.2)

in the corresponding operator norm. It guarantees that u is the unique solution of (1.3) and that the standard iteration (1.2) converges to u for every starting point u0 at a linear rate:

$$ \|u_{m+1}-u\| \leq\zeta\|u_{m}-u\|. $$

Note that ζ is a property of the chosen iteration (1.2) and of the chosen norm.Footnote 1

It will be convenient to use the starting value

$$ u_{0}=0 $$

throughout the paper. It leads to the relative error

$$ \frac{\|u-u_{m}\|}{\|u\|}\leq\zeta^{m} $$
(2.3)

after m steps of the iteration, and hence the number of steps needed for an ε-solution is upper bounded by

$$ m_{(1.2)}(\varepsilon)=\left\lceil \frac{\ln\varepsilon}{\ln\zeta}\right\rceil. $$
(2.4)

Recall that it is always assumed that 0 < ε ≤ 1.

We now consider the modified iteration (1.5) in which in the (m + 1)-th step Nf is replaced by some (simpler) approximation gm+ 1. A first general statement on this approach is the following.

Proposition 2.1

Let u be a solution to (1.3) and assume (2.2). If \(g_{m}\rightarrow Nf\), then for the iterates (1.5), \(\hat u_{m}\rightarrow u\).

Proof

Let ε > 0. For some mε we have ∥Nf − gm+ 1∥≤ ε for all m ≥ mε. Taking the difference of (1.3) with (1.5) gives

$$ u-\hat u_{m+1}=M(u-\hat u_{m}) + Nf - g_{m+1}. $$
(2.5)

Hence, the error \(\hat \eta _{m}:=\|u-\hat u_{m}\|\) satisfies

$$ \hat \eta_{m+1}\leq\zeta\hat\eta_{m}+{\varepsilon} \qquad \text{for all $m\geq m_{{\varepsilon}}$.} $$
(2.6)

Define the recursive sequence \(\hat \eta _{m+1}^{\prime } = \zeta \hat \eta _{m}^{\prime } + {\varepsilon }\) by fixing \(\hat \eta _{m_{{\varepsilon }}}^{\prime } = \hat \eta _{m_{{\varepsilon }}}\). Then \(\hat \eta _{m}^{\prime } \to {\varepsilon } / (1 - \zeta )\) and hence

$$ \limsup_{m \to \infty} \hat \eta_{m} \le \limsup_{m \to \infty} \hat \eta_{m}^{\prime} = \frac{{\varepsilon}}{1 - \zeta} $$

by (2.6). Since ε was arbitrary, this proves the assertion. □

In the following, we assume that the gm approximate Nf exponentially fast and satisfy

$$ \|Nf - g_{m}\| \leq C \|Nf\| \xi^{m},\qquad\text{where }\xi\leq\zeta $$
(2.7)

and C > 0 is a constant (that may depend on Nf). Then the error after m steps of the modified iteration (1.5) can be estimated. As for the standard iteration it will be convenient to consider here and in the following only the starting point

$$ \hat u_{0} = 0 $$

for our estimates.

Proposition 2.2

Let u be a solution to (1.3). Assume (2.2) and (2.7). Then it holds for the modified iteration (1.5) with starting point \(\hat u_{0} = 0\) that

$$ \|u-\hat u_{m}\| \leq\zeta^{m}\|u\|+C\|Nf\| \sum\limits_{\ell=0}^{m-1}\zeta^{\ell}\xi^{m-\ell}=\zeta^{m}\|u\|+\zeta^{m} C\|Nf\| \left\{\begin{array}{ll} \frac{1-(\xi/\zeta)^{m}}{(\zeta/\xi) - 1} &~\text{ if }\xi<\zeta,\\ m &~\text{ if }\xi=\zeta. \end{array}\right. $$

The proof is an immediate induction from (2.5) and (2.7). Now note that due to u − Mu = Nf it holds that

$$ (1-\zeta) \|u\| \leq \|Nf\| \leq (1+\zeta) \|u\|. $$

From Proposition 2.2 we thus obtain the following estimate on the relative error for the modified iteration with starting point \(\hat u_{0} = 0\):

$$ \frac{\|u - \hat u_{m}\|}{\|u\|} \leq \zeta^{m} + \zeta^{m}C(1 + \zeta) \left\{\begin{array}{ll} \frac{1-(\xi/\zeta)^{m}}{(\zeta/\xi) - 1} &~\text{ if }\xi<\zeta,\\ m &~\text{ if }\xi=\zeta. \end{array}\right. $$
(2.8)

This should be compared with (2.3).

Example 2.3

If we choose the gm+ 1 such that C = 1 and ξ = ζ/2 in (2.7), then (2.8) implies

$$ \frac{\|\hat u_{m}-u\|}{\|u\|}\leq \zeta^{m} (2 + \zeta). $$

To obtain a relative error ε, the modified iteration (1.5) hence requires at most

$$ m_{(1.5)}(\varepsilon)=\left\lceil \frac{\ln\varepsilon- \ln(2+\zeta)}{\ln\zeta}\right\rceil $$
(2.9)

steps. The standard iteration (1.2) needs only \(m_{(1.2)}(\varepsilon ) = \smash {\lceil \frac {\ln \varepsilon }{\ln \zeta }\rceil }\) steps. Note, however, that \(\ln (2 + \zeta ) < \ln (3) <1.1\). Therefore in the case of a fast standard iteration, when \(\ln \zeta \) is not close to zero, the number of additional steps in the modified iteration is very small. For instance with \(\zeta =\frac {1}{2}\) the additional term \(\frac {\ln (2+\zeta )}{|\ln \zeta |}\) equals 1.322, so the modified iteration needs at most two steps more than the standard iteration.

Generalizing this example we can derive a bound for the required number of steps for reaching a certain accuracy by estimating the inverse function of the right-hand side in (2.8). In the worst case ξ = ζ this requires some effort. We have the following result.

Proposition 2.4

Let u be a solution to (1.3). Assume (2.2) and (2.7). For 0 < ε ≤ 1, let m(1.5)(ε) be the number of steps needed for the modified iteration (1.5) to reach an ε-solution satisfying (2.1). In case ξ < ζ it holds that

$$ m_{(1.5)}(\varepsilon) \le\left\lceil \frac{\ln\varepsilon}{\ln\zeta} + K_{1}(\zeta,\xi,C) \right\rceil, \qquad K_{1}(\zeta,\xi,C) = \frac{\ln\left( 1 + \frac{C(1+\zeta)}{(\zeta/\xi) - 1} \right)}{|\ln\zeta|}, $$
(2.10)

whereas in case ξ = ζ it holds that

$$ m_{(1.5)}(\varepsilon) \le \left\lceil {\frac{\ln\varepsilon}{\ln\zeta} + K_{2}(\zeta,C) + \sqrt{\frac{2}{\ln\zeta^{-1}}} \sqrt{ \frac{\ln\varepsilon}{\ln\zeta} + K_{2}(\zeta,C) + \frac{1}{\ln\zeta} + \frac{1}{C(1+\zeta)}}}\right\rceil, $$
(2.11)

with

$$ K_{2}(\zeta,C) = \frac{\ln\ln(\zeta^{-1/(C(1+\zeta))}) }{\ln\zeta} = \frac{|\ln\ln\zeta^{-1}| + \ln(C(1+\zeta))}{|\ln\zeta|}. $$

The proof for ξ < ζ simply follows from (2.8) by omitting the term (ξ/ζ)m and rearranging for m. The case ξ = ζ is treated as Lemma A.1 in the Appendix, where also the accuracy of the bound (2.11) is illustrated in Fig. 4. It shows that the estimate is reasonably good, but too pessimistic for ζ close to one. Both constants K1(ζ, ξ, C) and K2(ζ, C) are unbounded for ζ → 1.

Recall that \(m_{(1.2)}(\varepsilon )=\lceil \frac {\ln \varepsilon }{\ln \zeta } \rceil \) is the iteration bound for an ε-solution with the standard iteration. If ζ/ξ is sufficiently large, then (2.10) shows that the number of additional steps required by the modified iteration for reaching the same accuracy is effectively constant, and indeed small if ζ itself is very small, see Example 2.3. In the case ξ = ζ we can roughly state that

$$ m_{(1.5)}(\varepsilon) \le m_{(1.2)}(\varepsilon) + K_{2}(\zeta,C) + O\left( \sqrt{m_{(1.2)}(\varepsilon) + K_{2}(\zeta,C)}\right), $$

but with a constant that behaves like \(1/ \ln \zeta ^{-1}\) when ζ → 1. In practice, for a fixed ζ, say up to ζ ≤ 0.9, and reasonable ε, the actual number of additional steps asserted by this bound is still effectively constant, as can be seen in Fig. 4 in the Appendix. For example, for a fast iteration with \(\zeta = \frac {1}{2}\) and C = 1, (2.11) provides the bound

$$ m_{(1.5)}(\varepsilon) \le \left\lceil \frac{|\ln\varepsilon| + 0.772 + \sqrt{2|\ln\varepsilon| + 0.469}}{\ln2} \right\rceil $$

for the case ξ = ζ. For small ε this is considerably worse than (2.9), where ξ = ζ/2, but in turn this bound is actually valid for all possible \(\xi \le \zeta =\frac {1}{2}\).

We conclude this section by mentioning a further possible modification of the standard linear iteration, in which instead of a fixed iteration matrix M a sequence \(M_{m}\rightarrow M\) is used. This leads to iterations of the form

$$ \bar u_{m+1}=M_{m+1}\bar u_{m} + g_{m+1}. $$

The matrix Mm could be implicitly given by a fixed linear iterative solver applied to a family of approximations Am → A of the linear system itself, or by a sequence Nm → N of preconditioners. Assuming (2.2), it is not difficult to prove that if gm → Nf and Mm → M, then \(\bar u_{m}\rightarrow u\), the fixed point of (1.3), the argument is similar to the proof of Proposition 2.1. Based on suitable assumptions on the convergence speed Mm → M one can then study error estimates. In this work, however, we restrict our attention to the simpler variation (1.5) with M being fixed.

3 Rank Growth in the Standard and Modified Iteration

The modified iteration (1.5) will usually need some more steps than the standard iteration (1.2) to reach a target accuracy ε for the relative error, which is indicated by the error estimates stated in the previous section. In turn the rank of the iterates may grow a little bit less per step, since we are adding gm+ 1 instead of Nf. In this section we compare the achievable accuracy with the accumulated representation ranks of the approximate solutions generated by the standard iteration and its modification in simplified model cases. While the results are perhaps too generic (and thereby too pessimistic) to use when studying a particular linear equation, our aim is to show that the modified iteration can provide some improvement. We mention again that rank truncation during the iteration is not considered in our analysis, but recommended in computations. The required ranks for a certain accuracy in practice hence can be much smaller than the rank bounds obtained below.

3.1 Rank Growth in the Standard Iteration

Due to the representation

$$ u_{m} = \left( \sum\limits_{\ell=0}^{m-1}M^{\ell}\right) Nf, $$
(3.1)

the ranks for the iterates of the standard iteration (with u0 = 0) can be estimated in terms of the following, in general unknown, constants:

$$ \nu_{m}= \nu_{m}(M) = \underset{v \neq 0}{\sup} \frac{\text{ rank}\left( {\sum}_{\ell = 0}^{m-1} M^{\ell} v \right)}{\text{ rank}(v)}. $$

Using these constants we have the following obvious estimate from (3.1).

Proposition 3.1

Consider the standard linear fixed-point iteration (1.2) with starting point u0 = 0. Then

$$ \text{rank}(u_{m})\leq \nu_{m} \text{ rank}(Nf). $$
(3.2)

In general all the νm could be infinite or equal to the dimension of V. A basic assumption in our paper is that at least for small m the νm are small compared to the dimension of V and do not grow too fast. But even then, since in the definition of νm we have taken a supremum, the estimate (3.2) is quite generic and our results will not account for any additional structure that could be exploited when applying powers of M to Nf in a particular instance. Another issue is that the estimate (3.2) is only reasonable if rank(Nf) is finite. Let us assume this, then together with the convergence speed (2.4) one obtains rank bounds for an ε-solution of the fixed-point equation (1.3), depending on the behaviour of the constants νm.

When the constants νm are not known, it is possible to estimate them by the constants

$$ \mu_{\ell} = \mu_{\ell}(M) = \underset{v \neq 0}{\sup} \frac{\text{ rank}(M^{\ell} v)}{\text{ rank}(v)}, $$

which can be easier to determine. In most cases we may rightfully assume

$$ \mu_{1} > 1. $$

Then clearly,

$$ \mu_{\ell}\le\mu_{1}^{\ell}, $$

and therefore

$$ \nu_{m} \le \sum\limits_{\ell = 0}^{m-1} \mu_{\ell} \le \sum\limits_{\ell = 0}^{m-1} \mu_{1}^{\ell}. $$
(3.3)

We call the upper bound in this estimate the worst-case behaviour, since in the typical case μ1 > 1 it indicates exponential rank growth in the standard iteration. It leads to rather pessimistic results.

Example 3.2

Consider the worst-case behaviour (3.3) with μ1 > 1. Then (3.2) yields

$$ \text{ rank}(u_{m})\leq\left( \frac{{\mu_{1}^{m}}-1}{\mu_{1}-1}\right)\text{ rank}(Nf). $$

With (2.4) it implies that for ε > 0, there exists an ε-solution uε for the linear equation (1.1) satisfying (2.1) and with a rank bounded by

$$ \text{ rank}(u_{\varepsilon})\leq\left( \frac{\mu_{1}^{\lceil\frac{\ln\varepsilon}{\ln\zeta}\rceil}-1}{\mu_{1}-1}\right) \text{ rank}(Nf)=O\left( \varepsilon^{\frac{\ln\mu_{1}}{\ln\zeta}}\right)\text{ rank}(Nf) $$

for \(\varepsilon \rightarrow 0\). If Nf has finite rank we can deduce an algebraic decay rate for the best low-rank approximation error of u with respect to the rank, namely

$$ \tau_{r}(u)=O\left( r^{\frac{\ln\zeta}{\ln\mu_{1}}}\right) $$

for \(r\rightarrow \infty \), where τr are the approximation numbers defined in (1.6).

There exist interesting examples for which the νm do not increase exponentially. As mentioned in the introduction, for sparse approximation in \(\mathbb {R}^{n}\) (rank being number of nonzero elements) a banded matrix M with bandwidth 1 + b will increase the number of nonzero elements of a vector by at most a factor μ1 ≤ 1 + b. However, it holds μℓ ≤ 1 + ℓb, since Mℓ has bandwidth 1 + ℓb. Indeed, the band support for different powers of M is nested so that

$$ \nu_{m} \le 1 + (m-1)b. $$

As another example, assume M is of the form

$$ M = M_{1} + M_{2}, $$

where both M1 and M2 do not increase the rank when applied to any u, that is, μ1(M1) ≤ 1 and μ1(M2) ≤ 1. Assume furthermore that M1 and M2 commute. Then

$$ M^{\ell}= \sum\limits_{k=0}^{\ell} \left( \begin{array}{cc}{\ell}\\ {k} \end{array}\right){M_{1}^{k}} M_{2}^{\ell-k} $$

shows that in such a case we have

$$ \mu_{\ell}\le\ell+1. $$

This implies

$$ \nu_{m} \le \frac{m(m+1)}{2}. $$

However, in special cases one can go further. Assume additionally that p(M2) is rank-preserving for any polynomial p. Then

$$ \sum\limits_{\ell = 0}^{m-1} M^{\ell}= \sum\limits_{\ell = 0}^{m-1}\sum\limits_{k=0}^{\ell}\left( \begin{array}{cc}{\ell}\\ {k} \end{array}\right) {{M}_{1}^{k}} {M}_{2}^{\ell-k} = \sum\limits_{k=0}^{m-1} {{M}_{1}^{k}} \left( \sum\limits_{\ell = k}^{m-1} \left( \begin{array}{cc}{\ell}\\ {k} \end{array}\right) M_{2}^{\ell-k}\right) $$

implies

$$ \nu_{m} \le m. $$
(3.4)

For matrix equations an operator of the form,

$$ M = \tilde{M}_{1} \otimes \tilde{M}_{2} + I \otimes \tilde{M}_{3}, $$

has the considered properties, provided \(\tilde M_{2}\) and \(\tilde M_{3}\) commute. This includes the Kronecker product operators (\(\tilde M_{3} = 0\)) and Sylvester-type operators (\(\tilde M_{2} = I\)). Both examples can be generalized to operators of such form on tensor spaces, but in the case of the Sylvester-like structure, νm becomes a polynomial of higher order.

Example 3.3

If the linear rank growth (3.4) is assumed, then (3.2) becomes

$$ \text{rank}(u_{m})\leq m \text{ rank}(Nf). $$

Hence, if rank(Nf) is finite, an ε-solution exists satisfying

$$ \text{rank}(u_{\varepsilon})\leq \left\lceil\frac{\ln\varepsilon}{\ln\zeta}\right\rceil \text{ rank}(Nf). $$

It implies a super-algebraic decay rate

$$ \tau_{r}(u)=O(\zeta^{r}) $$

for the best rank-r approximation error of the solution u to a fixed-point equation (1.3).

More generally, for a polynomial growth νm ≤ p(m), where p is a polynomial of degree q, one obtains \(\text {rank}(u_{\varepsilon }) = O((\ln \varepsilon /\ln \zeta )^{q})\) and \(\tau _{r}(u) = O(\zeta ^{(r^{1/q})})\).

3.2 Standard Iteration with Fixed Approximation of N f

Now we discuss the case that Nf has very large or infinite rank. In practice, when an approximation

$$ \frac{\|Nf-N\tilde{f}\|}{\|Nf\|}\leq\delta $$
(3.5)

is available, where \(N\tilde {f}\) has finite rank, we can simply use \(N\tilde {f}\) in the standard iteration. This is equivalent to solving a perturbed fixed-point equation

$$ v=Mv+N\tilde{f}, $$
(3.6)

which in case that N is invertible corresponds to a linear equation

$$ Av=\tilde{f} $$

instead of (1.1). The corresponding standard iteration reads

$$ v_{m+1}=Mv_{m}+N\tilde{f},\qquad v_{0}=0, $$
(3.7)

and converges to \(v=(I-M)^{-1}N\tilde {f}\). The relative error to the original fixed point u = (I − M)− 1Nf can be estimated as follows:

$$ \frac{\|v_{m}-u\|}{\|u\|} \leq \frac{\|u-v\|}{\|u\|} + \frac{\|v-v_{m}\|}{\|\tilde{u}\|}\cdot \frac{\|v\|}{\|u\|} \leq \left( \frac{1+\zeta}{1-\zeta}\right)\delta + \frac{\|v-v_{m}\|}{\|v\|}\left[1+\left( \frac{1+\zeta}{1-\zeta}\right)\delta\right]. $$
(3.8)

For simplicity, let us choose target accuracies of the form

$$ \delta\leq\left( \frac{1-\zeta}{1+\zeta}\right) \frac{\varepsilon}{2+\varepsilon},\qquad\frac{\|v-v_{m}\|}{\|\tilde{u}\|} \leq \frac{\varepsilon}{2} $$
(3.9)

for (3.5) and (3.7). Then (3.8) becomes

$$ \frac{\|v_{m}-u\|}{\|u\|}\leq\varepsilon, $$
(3.10)

where vm satisfies

$$ \text{ rank}(v_{m})\leq \nu_{m} \text{ rank}(N\tilde{f}) $$
(3.11)

by Proposition 3.1. The second inequality in (3.9) is satisfied after at most

$$ m=\left\lceil \frac{\ln\varepsilon-\ln2}{\ln\zeta}\right\rceil $$

iterations. One can now proceed as above by assuming different cases for νm and \(\text {rank}(N\tilde {f})\).

Example 3.4

For later comparison with the modified iteration, we assume (2.7) holds with C = 1 and rank(gm) ≤ m0 ⋅ m. Then we choose \(N\tilde {f}=g_{\tilde {m}}\) such that (3.5) is satisfied for \(\delta =\left (\frac {1-\zeta }{1+\zeta }\right ) \frac {\varepsilon }{2+\varepsilon }\) as required in (3.9). For this, we need to truncate (2.7) after \(\tilde {m}=\lceil \frac {\ln \delta }{\ln \xi }\rceil \) terms so that

$$ \text{rank}(N\tilde{f})\leq m_{0}\cdot\left\lceil \frac{\ln\delta}{\ln\xi}\right\rceil = m_{0}\cdot\left\lceil \frac{\ln\varepsilon+\kappa(\zeta,\varepsilon)}{\ln\xi}\right\rceil,\qquad\kappa(\zeta,\varepsilon):=\ln\left( \frac{1-\zeta}{(2+\varepsilon)(1+\zeta)}\right). $$

Assuming the worst case (3.3) of exponential rank growth, we conclude from (3.10) and (3.11) that there exists an ε-solution uε for the initial fixed-point equation (1.3) that satisfies

$$ \text{rank}(u_{\varepsilon}) \leq \left( \frac{\mu_{1}^{\left\lceil \frac{\ln\varepsilon - \ln 2}{\ln\zeta}\right\rceil} -1}{\mu_{1}-1}\right)\text{ rank}(N\tilde f) = O\left( \mu_{1}^{\frac{\ln 2}{\ln \zeta^{-1}}} \varepsilon^{\frac{\ln \mu_{1}}{\ln \zeta}} \left( \frac{\ln \varepsilon + \kappa(\zeta,\varepsilon)}{\ln \xi} \right) \right), $$
(3.12)

where the constant only depends on μ1.

Correspondingly, in case of linear rank growth μm ≤ m, we obtain

$$ \text{rank}(u_{\varepsilon}) \leq \left\lceil \frac{\ln\varepsilon- \ln2}{\ln\zeta}\right\rceil \text{ rank}(N\tilde f) = O\left( \left( \frac{\ln\varepsilon}{\ln\zeta} \right) \left( \frac{\ln\varepsilon+ \kappa(\zeta,\varepsilon)}{\ln\xi} \right)\right). $$
(3.13)

The bounds (3.12) and (3.13) are depicted in Figs. 1 and 2 further below for some values of ζ and ξ.

Fig. 1
figure 1

Rank bounds for ε-solutions with the modified iteration (1.5) and exponential growth μℓ ≤ 2ℓ, assuming (3.16) and (3.17). Left: ζ = 0.5, right: ζ = 0.9. Solid lines: rank bound (3.24) using the minimal m(1.5)(ε) (solid lines in Fig. 4). Dotted lines: rank bound for a modified iteration (3.26) with target accuracy \(N \tilde f = g_{\tilde m}\) according to (3.28). Cross and plus markers: standard iteration using the same \(N \tilde f\) according to (3.12) for ξ = ζ/2 (cross) and ξ = ζ (plus)

Full size image
Fig. 2
figure 2

Rank bounds for ε-solutions with the modified iteration and linear growth μℓ ≤ ℓ + 1. Left: ζ = 0.5, right: ζ = 0.9. Solid lines: rank bound (3.25) using the minimal m(1.5)(ε) (solid lines in Fig. 4). Dotted lines: rank bound for a modified iteration (3.26) with target accuracy \(N \tilde f = g_{\tilde m}\) according to (3.29). Cross and plus markers: standard iteration using the same \(N \tilde f\) according to (3.13) for ξ = ζ/2 (cross) and ξ = ζ (plus)

Full size image

3.3 Rank Growth in the Modified Iteration

In the modified iteration (1.5) we can deal with the case that Nf has large rank by replacing it with a sequence gm with growing ranks. In non-recursive form, the modified iteration with \(\hat u_{0} = 0\) reads

$$ \hat u_{m} = M^{m-1} g_{1} + M^{m-2} g_{2} + \dots+ M^{0} g_{m}. $$

This can also be written as

$$ \hat u_{m} = \left( \sum\limits_{\ell = 0}^{m-1} M^{\ell}\right) g_{1} + \left( \sum\limits_{\ell = 0}^{m-2} M^{\ell}\right)(g_{2} - g_{1}) + {\dots} + M^{0}(g_{m} - g_{m-1}). $$

Instead of Proposition 3.1 we hence have the following rank estimates.

Proposition 3.5

Consider the modified iteration (1.5) with starting point \(\hat u_{0} = 0\). Then

$$ \text{rank}(\hat u_{m}) \le \sum\limits_{\ell= 0}^{m-1} \mu_{\ell}\text{ rank}(g_{m - \ell}) $$
(3.14)

and

$$ \text{ rank}(\hat u_{m}) \le\sum\limits_{\ell = 1}^{m} \nu_{\ell} \text{~rank}(g_{m-\ell+1} - g_{m - \ell}), $$
(3.15)

where g0 = 0.

For the standard iteration, knowing the behaviour of the constants νm or μℓ is sufficient for deriving approximation results in terms of ζ. In the case of the modified iteration we also need to know how fast the ranks of gm grow in relation to how fast the error ∥Nf − gm∥ tends to zero. We keep the assumption (2.7), but restrict to C = 1, that is,

$$ \|Nf - g_{m} \| \le\| Nf \| \xi^{m} $$
(3.16)

for some ξ ≤ ζ, and consider the simplest case that the rank of the gm grow linearly, that is,

$$ \text{rank}(g_{m}) \le m_{0} \cdot m, $$
(3.17)

where m0 is a fixed constant. In combination with (2.7) this assumption is equivalent to Nf belonging to a certain approximation class defined by

$$ \tau_{r} (Nf) \lesssim\xi^{r}, $$

where τr again are the approximation numbers (1.6). Note however that in a practical method the gm must be available. When the rank function is defined by a dictionary \(\mathcal D\), a most reasonable model for (3.17) is that Nf admits an initial expansion

$$ Nf = \sum\limits_{i=1}^{R} h_{i}, \qquad h_{i}\in\mathcal{D}, $$
(3.18)

and then approximating it by batches of m0 terms taking

$$ g_{m} = \sum\limits_{j = 1}^{m} \left( h_{(j-1) m_{0} + 1} + \cdots+ h_{j m_{0}}\right). $$
(3.19)

In this case

$$ \text{rank}(g_{m} - g_{m-1}) \le m_{0} $$
(3.20)

for all m. A related approach that arises in practice is that a dictionary expansion of \(f = {\sum }_{i} f_{i}\) is given. Then assuming that the operator N does not increase rank by more than a factor μN, one could take \(g_{m} = N (f_{1} + {\dots } + f_{m})\) so that rank(gm − gm− 1) ≤ μN for all m.

Clearly in (3.19) we can trade a larger batch size m0 for a faster approximation rate. In general, if (3.16) and (3.17) hold for some sequence gm one can define for an integer t > 1 the sequence

$$ {g}_{m}^{\prime}=g_{t\cdot m}, $$
(3.21)

which will satisfy (3.16) and (3.17), too, but with different constants \(\xi ^{\prime } = \xi ^{t}\) and \(m_{0}^{\prime }=tm_{0}\). In particular, in the case that ξ = ζ we can always pass to \(\xi ^{\prime } = \zeta ^{2}<\zeta \) which enables the more accurate estimate (2.10) for the required number of steps in Proposition 2.4. The difference will be illustrated in some numerical comparisons further below.

With (3.17) the rank estimate (3.14) simplifies to

$$ \text{rank}(\hat u_{m}) \le m_{0} \sum\limits_{\ell= 0}^{m-1} \mu_{\ell}(m - \ell). $$
(3.22)

If also (3.20) holds, (3.15) simplifies to

$$ \text{rank}(\hat u_{m}) \le m_{0} \sum\limits_{\ell = 1}^{m} \nu_{\ell}. $$
(3.23)

We next consider the same two examples for the behaviour of νm as in Section 3.1.

Example 3.6

As in Example 3.2 assume the worst-case scenario (3.3) of exponential growth. In this case both simplified bounds (3.22) and (3.23) yield

$$ \text{rank}(\hat u_{m}) \le m_{0} \left( \frac{\mu_{1}}{\mu_{1} - 1} \right)\left( \frac{{{\mu}_{1}^{m}} - 1 - m + \frac{m}{\mu_{1}}}{\mu_{1} - 1} \right). $$
(3.24)

For rigorous bounds on the rank of an ε-solution we can insert the estimates for m(1.5)(ε) provided in (2.10) and (2.11) in the right-hand side of (3.24). We omit the resulting formulas. The asymptotic behaviour is \(\text {rank}(u_{\varepsilon })\lesssim \mu _{1}^{\frac {\ln \varepsilon }{\ln \zeta }} \sim \varepsilon ^{\frac {\ln \mu _{1}}{\ln \zeta }}\) when ξ < ζ, but with a constant deteriorating for \(\xi \rightarrow \zeta \) and \(\zeta \rightarrow 1\). It implies again \(\tau _{r}(u)=O\left (r^{\frac {\ln \zeta }{\ln \mu _{1}}}\right )\) with corresponding constants. If ξ = ζ the estimate (2.11) technically yields \(\text {rank}(u_{\varepsilon })\lesssim \mu _{1}^{\frac {\ln \varepsilon }{\ln \zeta }+\sqrt {\frac {\ln \varepsilon }{\ln \zeta }}} \sim \varepsilon ^{\frac {\ln \mu _{1}}{\ln \zeta }\left (1+\sqrt {\frac {\ln \zeta }{\ln \varepsilon }}\right )}\) (with a constant deteriorating for \(\zeta \rightarrow 1\)), but as explained above, in the considered model (3.16) and (3.17) we can always assume ξ = ζ2 < ζ. Figure 1 further below contains the precise bounds for some combinations of ζ and ξ.

Example 3.7

If we proceed with the bound (3.23) and assume a linear rank growth, νℓ ≤ ℓ, as in Example 3.3, then we get the rank estimate

$$ \text{rank}(\hat u_{m}) \le m_{0} \left( \frac{m (m+1)}{2} \right). $$
(3.25)

From (2.10) and (2.11) we then obtain the asymptotic bound

$$ \text{rank}(u_{\varepsilon})\lesssim\left( \frac{\ln\varepsilon}{\ln\zeta}\right)^{2} $$

for an ε-solution of (1.3), with different constants for the cases ξ < ζ and ξ = ζ. The constants deteriorate with \(\xi \rightarrow \zeta \) and \(\zeta \rightarrow 1\), respectively. Some concrete values are plotted in Fig. 2 below. We omit more detailed formulas and just note the implied approximation rate \(\tau _{r}(u)=O(\zeta ^{\sqrt {r}})\) for \(r\rightarrow \infty \), with constants depending on ζ, ξ and m0.

3.4 Modified Iteration with Target Accuracy for N f

Since one seeks only for an ε-solution to the fixed-point equation (1.3) it may not be necessary to approximate Nf with higher and higher rank. Similar to the discussion for the standard iteration in Section 3.2, one can terminate at some \(g_{\tilde {m}}=N\tilde {f}\) satisfying \(\|Nf-g_{\tilde {m}}\|\leq \delta \|Nf\|\) as in (3.5) and then proceed with \(g_{m}=g_{\tilde {m}}\) for all subsequent iterations.

We can analyse such an approach as a modified iteration

$$ \hat{v}_{m+1}=M\hat{v}_{m}+\tilde{g}_{m+1} $$
(3.26)

for the perturbed fixed-point equation \(v=Mv+N\tilde {f}\) as in (3.6), where we use \(\tilde {g}_{m+1}=g_{m+1}\) for \(m=1,\dots ,\tilde {m}-1\) as approximations of \(N\tilde {f}\), and \(\tilde {g}_{m+1}=g_{\tilde {m}}=N\tilde {f}\) for \(m\geq \tilde {m}\). Hence the first \(\tilde {m}\) iterates are identical to the modified iteration with gm. The competitor for this strategy is the standard iteration (3.7) that uses \(N\tilde {f}\) from the start. Since the error analysis (3.8) remains valid (with \(\hat v_{m}\) instead of vm), we can aim at the same target accuracies (3.9) (with \(\hat v_{m}\) instead of vm) as the standard iteration for guaranteeing an ε-solution for the initial fixed-point equation. If (3.16) holds, this means we have to take \(\smash {\tilde {m}=\lceil \frac {\ln \delta }{\ln \xi }\rceil }\) as in Example 3.4. We can expect a similar estimate as (3.16) for \(N\tilde {f}=g_{\tilde {m}}\), that is,

$$ \|N\tilde{f}-\tilde{g}_{m}\| \leq \tilde{C}\|N\tilde{f}\|\xi^{m}, $$
(3.27)

where \(\tilde {C}\) is some not too large constant. For example, we may assume the hi in a dictionary expansion (3.18) of Nf to be pairwise orthogonal, as would be the case in sparse approximation in \(\mathbb {R}^{n}\) or in a singular value decomposition of a matrix. Then we can take \(\tilde {C}=(1-\xi ^{2\tilde {m}})^{-1/2}\), since \(\|N\tilde {f}-g_{m}\| \leq \|Nf-g_{m}\| \leq \|Nf\|\xi ^{m}\) and

$$ \frac{\|\tilde{N}f\|^{2}}{\|Nf\|^{2}} \geq 1-\frac{\|Nf-g_{\tilde{m}}\|^{2}}{\|Nf\|^{2}}\geq1-\xi^{2\tilde{m}}, $$

which yields (3.27) for \(m \le \tilde m\) (for larger m the left side of (3.27) is zero anyway).

Let \(\hat m\) be the number of steps required for (3.26) to reach an (ε/2)-solution for v, which by (3.8) will be an ε-solution for the original u. Assuming \(\hat m =\tilde {m}+k\geq \tilde {m}\),Footnote 2 then according to (3.15) the final rank estimate will be

$$ \text{rank}(\hat{v}_{\hat m})\leq \sum\limits_{\ell = k + 1}^{\hat m} \nu_{\ell}\text{ rank}(g_{\hat m-\ell+1} - g_{\hat m - \ell}). $$

Example 3.8

For exponential rank growth (3.3), and assuming the model (3.20), this means

$$ \text{rank}(\hat v_{\hat m}) = m_{0} \left( \frac{\mu_{1}}{\mu_{1} - 1} \right) \left( \frac{{\mu}_{1}^{\hat m} - {{\mu}_{1}^{k}} - \tilde m + \frac{\tilde m}{\mu_{1}} }{\mu_{1} - 1} \right). $$
(3.28)

In the case of linear rank growth νm ≤ m one gets

$$ \text{rank}(\hat v_{\hat m}) \le m_{0} \left( \frac{ \hat m (\hat m+1) - k (k+1)}{2} \right). $$
(3.29)

We omit the formulas for rank estimates in terms of target accuracy ε. Numerical values are provided in Figs. 1 and 2. They indicate that using the modified iteration until some \(g_{\tilde m} = N \tilde f\) as derived above can outperform the standard iteration with fixed \(N \tilde f\).

3.5 Numerical Illustration of Error Bounds

In the numerical illustrations in Figs. 1 and 2 we compare the derived rank estimates for achieving an ε-solution with the modified iteration in the two scenarios of an exponential growth νm ≤ 2m − 1, i.e. μ1 = 2 in (3.3), and for a linear rank growth νm ≤ m.

Both scenarios are evaluated for the values ζ = 0.5 and ζ = 0.9 (spectral norm of M). We consider the approximation rate (2.7) for Nf with C = 1 in the two cases ξ = ζ/2 and ξ = ζ in (3.16), where we assume m0 = 1 in (3.20), that is, gm is obtained from gm− 1 by a rank-one update. To check the potential merit of a larger batch size in the case ξ = ζ, we also consider m0 = 2 (rank-two updates) with squared rate ξ = ζ2 (i.e. t = 2 in (3.21)). According to the plots, the larger batch size can be slightly beneficial for the case of exponential rank growth, but does not help in the case of linear rank growth. The following functions are shown in Figs. 1 and 2:

  • The rank bounds (3.24) (in Fig. 1) and (3.25) (in Fig. 2) for the modified iteration as solid lines, when using as m the minimal number of steps m(1.5)(ε) such that the right-hand side of (2.8) is less than ε. The values for m(1.5)(ε) are determined numerically and are depicted in Fig. 4 in the Appendix (as solid lines). One could use instead the derived upper bounds (2.10)–(2.11) for m(1.5)(ε) (these can be seen in in Fig. 4 as dotted lines). One then obtains slightly worse rank bounds, especially in the case ξ = ζ.

  • The rank bounds for a modified iteration (3.26) as dotted lines, using some truncation \(N \tilde f = g_{\tilde m}\) as a final approximation, according to (3.28) (Fig. 1) and (3.29) (Fig. 2). We used \(\tilde C = (1 - \xi ^{2 \tilde m})^{-1/2}\) in (3.27), as motivated above.

  • The rank bound for the standard iteration when using the same truncation \(N \tilde f = g_{\tilde m}\), according to (3.12) (Fig. 1) and (3.13) (Fig. 2), respectively. These are only given for batch size m0 = 1 and are depicted as cross markers for ξ = ζ/2 and plus markers for ξ = ζ.

As can be seen from both figures, modified iterations can perform equally well or better than the standard iteration with truncated right-hand side. Especially for the case of linear rank growth, the modified iterations with a target accuracy for Nf (dotted lines) seem to provide a reasonable improvement for ξ = ζ, in particular keeping in mind that they are more data-sparse. It appears that with exponential rank growth the modified iteration should not be terminated at a fixed \(g_{\tilde m} = N \tilde f\), and that it helps to take a larger batch size to ensure fewer steps. However, the rank bounds, especially for ζ = 0.9, (Fig. 1, right) are ridiculously large and only of theoretical interest. This illustrates that if for a given linear equation (1.1) there does not exist an iteration that is either fast or not exponentially rank increasing, its solution might not admit a good low-rank approximation.

4 Numerical Experiment

Finally, we include a small numerical experiment to compare the actual convergence of the methods for a particular problem. We generate a 1000 × 1000 tridiagonal matrix A = L + D + R, with diagonal entries in D uniformly distributed in the interval [2,3], whereas the lower and upper off diagonal entries in L and R are uniformly distributed in [− 2,− 1] and [− 1,0], respectively. The goal is to solve the linear equation Au = f, where f has exponentially decaying entries fi = (4/5)i. Since the exact solution can be well approximated by sparse vectors, we aim at iterations that build approximations with possibly few nonzero entries. Hence the rank function here is the number of nonzero entries in a vector.

We employ two iterative solvers (1.2), the Jacobi method, where N = D− 1, and an approximate Gauss–Seidel method, with

$$ N = D^{-1}\left( I - LD^{-1} + (LD^{-1})^{2}\right), $$

which is an approximation of (L + D)− 1 = D− 1(I + LD− 1)− 1. Correspondingly,

$$ M = I - NA $$

is a tridiagonal banded matrix for the Jacobi method (with zero diagonal), and a five-banded matrix (with only one upper diagonal) for the approximate Gauss–Seidel method. We use the standard iterations (1.2) and modified iterations (1.5). As approximations gm → Nf we take gm = Nfm, where fm contains the largest m entries of f in modulus. Compared to taking the m largest entries of Nf this has the advantage that the gm can be recursively computed from the sparse columns of N without forming Nf. (Almost no difference was observed for the two approaches.) All four resulting methods are also tested in a truncated version, where after every step entries smaller than a fixed threshold are deleted from the iterate.

The results for various instances of A varied slightly but were overall quite similar. In Fig. 3 we show one of the better outcomes. The left plot shows the decrease of the relative error (in Euclidean norm) to the exact solution u, with the dashed lines corresponding to the standard iteration and solid lines to the modified iteration. The numerically computed spectral radii and spectral norms in this instance were ρ(M) ≈ 0.934 and ζ = ∥M∥≈ 1.145 for the Jacobi method, and ρ(M) ≈ 0.885 and ζ = ∥M∥≈ 0.981 for the approximate Gauss–Seidel method. The threshold in the truncated versions was 10− 9 to reach a relative accuracy below 10− 8.

Fig. 3
figure 3

Numerical results for the solution of a tridiagonal linear system by the Jacobi method or an approximate Gauss-Seidel method (‘GS’). Dashed lines: standard iterations (with and without truncation). Solid lines: modified iterations (with and without truncation). The dotted black line on the right shows the best possible relative error for a given number of nonzero entries

Full size image

In the right panel of Fig. 3 we investigate the convergence speed with respect to the number of used nonzero entries. For the standard iterations only the truncated versions are shown (the vertical dashed lines), since without truncation the iterates immediately fill up. While all truncated methods eventually need about the same number of nonzero entries for a relative error in the magnitude of the threshold, the modified iterations need less nonzeros during the overall process. One should also recall that the standard methods operate with a full vector Nf throughout. Note that the Jacobi method without truncation (blue line), while being slower, is capable of constructing relatively sparse solutions, whereas the approximate Gauss–Seidel method (red line) clearly requires the truncation. For comparison, the right panel also displays the best possible (relative) sparse approximation errors (i.e. the decay of τr(u)) as a black dotted line, which are obtained from using the largest entries (in modulus) of the true solution u. The truncated approximate Gauss–Seidel method gets closest to this minimal error before reaching the number of non-zeros required for the accuracy specified by the threshold. Note that for both the truncated Jacobi and approximate Gauss–Seidel method the final error is essentially optimal with respect to the sparsity.