Foundations of Computational Mathematics

, Volume 17, Issue 4, pp 1037–1083 | Cite as

Iterative Methods Based on Soft Thresholding of Hierarchical Tensors

Article

Abstract

We construct a soft thresholding operation for rank reduction in hierarchical tensors and subsequently consider its use in iterative thresholding methods, in particular for the solution of discretized high-dimensional elliptic problems. The proposed method for the latter case adjusts the thresholding parameters, by an a posteriori criterion requiring only bounds on the spectrum of the operator, such that the arising tensor ranks of the resulting iterates remain quasi-optimal with respect to the algebraic or exponential-type decay of the hierarchical singular values of the true solution. In addition, we give a modified algorithm using inexactly evaluated residuals that retains these features. The effectiveness of the scheme is demonstrated in numerical experiments.

Keywords

Low-rank tensor approximation Hierarchical tensor format Soft thresholding High-dimensional elliptic problems 

Mathematics Subject Classification

41A46 41A63 65D99 65F10 65N12 65N15 

1 Introduction

Low-rank tensor decompositions have proven to be a very successful tool in the numerical approximation of high-dimensional problems, such as partial differential equations posed on high-dimensional domains. Such problems arise, for instance, in the context of multi-parametric, stochastic, or quantum-physical models; for an overview of various applications of structured tensor decompositions, we refer to the survey articles [25, 28, 34] and the references given there. In this work, we consider the task of finding approximations in low-rank form of high-dimensional functions defined implicitly as solutions of linear operator equations. This problem is not only of intrinsic interest, but also arises, for instance, in methods for time-dependent or eigenvalue problems.

Low-rank approximations can exploit particular structural features to achieve highly compressed representations of high-dimensional objects. However, this comes at the price of a strong degree of nonlinearity in these representations that makes the computation of such approximations a challenging problem. In the case of subspace-based tensor representations such as the hierarchical Tucker format [30] or the special case of the tensor train format [46], one has a notion of tensor rank as a tuple of matrix ranks of certain matricizations. From a computational perspective, these tensor formats have the major advantage that for any tensor given in such a representation, by a simple combination of linear algebra procedures, one may obtain an error-controlled, quasi-optimal approximation by a tensor of lower ranks. This is achieved by truncating the ranks of a hierarchical singular value decomposition [24, 47, 50], or HSVD for short, of the tensor.

We consider an alternative procedure for reducing ranks that is based on soft thresholding of the singular values in a HSVD, as opposed to the mentioned rank truncation (which would correspond to their hard thresholding). The new procedure has similar complexity and quasi-optimality properties, but unlike the truncation it is non-expansive. Making essential use of this feature, we construct iterative methods that are both guaranteed to converge and ensure strong tensor rank bounds for each iterate under very general conditions on the operator.

A large part of the results that we obtain are in fact applicable to fixed-point iterations based on general contractive mappings. The iterative scheme that we focus on as an example, however, is a Richardson iteration for solving the linear operator equation
$$\begin{aligned} {\mathcal {A}} \mathbf {u} = {\mathbf {f}} \end{aligned}$$
(1.1)
on a separable tensor product Hilbert space
$$\begin{aligned} {\mathcal {H}} = {\mathcal {H}}_1 \otimes \cdots \otimes {\mathcal {H}}_d, \end{aligned}$$
(1.2)
where \({\mathbf {f}} \in {\mathcal {H}}\) is given and \({\mathcal {A}}:{\mathcal {H}} \rightarrow {\mathcal {H}}\) is bounded, symmetric, and elliptic on \({\mathcal {H}}\), that is, the iteration is of the form
$$\begin{aligned} \mathbf{u}_{k+1} = {\mathbf {S}}_{\alpha _k} \bigl (\mathbf{u}_k - \omega ({\mathcal {A}} \mathbf{u}_k - {\mathbf {f}}) \bigr ), \end{aligned}$$
(1.3)
where \({\mathbf {S}}_{\alpha _k}\) is the proposed soft thresholding procedure with suitable parameters \(\alpha _k\).

Even when both \({\mathcal {A}}\) and \({\mathbf {f}}\) have exact low-rank representations, the unique solution \( \mathbf {u}^*\) of the problem (1.1) may no longer be of low rank. It turns out, however, that in many cases of interest, \(\mathbf {u^*}\) can still be efficiently approximated by low-rank tensors up to any given error tolerance. Here, one can obtain algebraic error decay with respect to the ranks under fairly general conditions [38, 49], and superalgebraic or exponential-type decay in more specific situations [1, 15, 23, 37].

When a solution \(\mathbf{u}^*\) that has this property is approximated by an iteration such as (1.3), it is not clear to what extent also the iterates \(\mathbf{u}_{k}\) retain comparably low ranks, since the basic iteration without truncation could in principle lead to an exponential rank increase. That the ranks of \(\mathbf{u}_k\) remain comparable to those needed for approximating \(\mathbf{u}^*\) at the current accuracy therefore depends essentially on the appropriate choice of thresholding parameters \(\alpha _k\). Keeping the tensor ranks of iterates as low as possible is of crucial importance for the computational complexity of such methods, since the number of operations for the procedures that need to be performed, such as orthogonalizations, grows like the fourth power of these ranks.

A fundamental question is therefore whether in the course of such an iteration, one can always fully profit from the low-rank approximability of \(\mathbf{u}^*\). In principle, the ranks of \(\mathbf{u}_k\) could depend, for instance, on the presence of a strong low-rank structure of \({\mathcal {A}}\), or on sufficiently good starting values \(\mathbf{u}_0\). We show in this work that when the rank reduction in each step is done by the soft thresholding procedure, quasi-optimal tensor ranks can be enforced for each single iterate\(\mathbf{u}_k\), independently of the rank increase caused by \({\mathcal {A}}\). Here, we may always start with rank zero, \(\mathbf{u}_0 = 0\). The corresponding \(\alpha _k\) are chosen by a simple a posteriori rule, requiring no information on the low-rank approximability of \(\mathbf{u}^*\). Our scheme thus automatically strikes a compromise, under quite general assumptions, between maintaining linear convergence of the iteration to \(\mathbf{u}^*\) and preventing the ranks of iterates from growing more strongly than necessary.

Throughout this paper, the notation \(A\lesssim B\) is used to indicate that there exists a constant \(C>0\) such that \(A \le C B\), and \(A \sim B\) if and only if \(A \lesssim B\) and \(B\lesssim A\).

1.1 Quasi-Optimality

By quasi-optimality of tensor ranks, we refer to the following property: Assuming that the hierarchical singular values of \(\mathbf{u}^*\) have a certain algebraic or exponential-type decay, the maximum tensor rank of each \(\mathbf{u}_k\) can be bounded, up to a uniform multiplicative constant, by the maximum hierarchical rank of the best hierarchical tensor approximation to \(\mathbf{u}^*\) of the same accuracy. To ensure this, we exploit the non-expansiveness of soft thresholding, which allows us to choose the thresholding parameters in each step as large as required to control the ranks, without compromising the convergence of the iteration.

We shall use weak-\(\ell ^p\) spaces to quantify algebraic decay of sequences. These spaces are defined as follows: For a given real sequence \(a = (a_k)_{k\in {\mathbb {N}}}\), for each \(n\in {\mathbb {N}}\), let \(a^*_n\) be the n-th largest of the values \(|a_k|\). Then, for \(p>0\), the space \(\ell ^{p,\infty }\) is defined as the collection of sequences for which
$$\begin{aligned} |a|_{\ell ^{p,\infty }} {:=} \sup _{n\in {\mathbb {N}}} n^{\frac{1}{p}} a^*_n \end{aligned}$$
is finite, and this quantity defines a (quasi-)norm on \(\ell ^{p,\infty }\); note that \(\ell ^p \subset \ell ^{p,\infty } \subset \ell ^{p'}\) for \( p < p'\). We will use these spaces with \(0<p<2\), where \(\ell ^{p,\infty } \subset \ell ^2\). One has \(a \in \ell ^{p,\infty }\) for such p precisely when there exists \(C>0\) such that
$$\begin{aligned} \left( \sum _{k>n} |a^*_k|^2 \right) ^\frac{1}{2} \le C n^{-s},\quad s {:=} \frac{1}{p} - \frac{1}{2}, \end{aligned}$$
(1.4)
and in this case, the infimum over such C is proportional to \(|a|_{\ell ^{p,\infty }}\), see, e.g., [17], that is, \(C = C_p |a|_{\ell ^{p,\infty }}\) where \(C_p>0\) depends only on p. In other words, the sequences in \(\ell ^{p,\infty }\) are precisely those for which the error of approximation in \(\ell ^2\) by the n entries largest in modulus decreases like \(n^{-s}\) or faster.

Let us now exemplify our rank estimates in the case \(d=2\), that is, for the low-rank approximation of \(\mathbf{u}\in \mathcal {H}_1\otimes \mathcal {H}_2\). Here, the notion of rank is the rank of the Hilbert–Schmidt operator from \(\mathcal {H}_2\) to \(\mathcal {H}_1\) that is induced by \(\mathbf{u}\), which reduces to the matrix rank in the case of finite-dimensional \(\mathcal {H}_1, \mathcal {H}_2\). For what follows, let \(\sigma =(\sigma _k)_{k\ge 1}\) be the corresponding singular values of \(\mathbf{u}\).

The error of best rank-n approximation of \(\mathbf{u}\) is then precisely \((\sum _{k>n} |\sigma _k|^2 )^\frac{1}{2}\). If \(\sigma \in \ell ^{p,\infty }\), then for the rank \(n_\varepsilon \) required to approximate \(\mathbf{u}\) up to an error \(\varepsilon \), by (1.4) we have the bound
$$\begin{aligned} n_\varepsilon \le \bigl (C_p|\sigma |_{\ell ^{p,\infty }}\bigr )^{\frac{1}{s}} \varepsilon ^{-\frac{1}{s}}, \end{aligned}$$
and this is the best bound we can obtain without further information on \(\mathbf{u}\). In this setting, we thus call a sequence \((\mathbf{v}_{k})_{k\in {\mathbb {N}}}\) of approximations of \(\mathbf{u}\) quasi-optimal if for \(\varepsilon _k\rightarrow 0\) we have
$$\begin{aligned} ||\mathbf{u}- \mathbf{v}_k|| \le \varepsilon _k , \qquad {{\mathrm{rank}}}(\mathbf{v}_k) \le \hat{C} |\sigma |_{\ell ^{p,\infty }}^{\frac{1}{s}} \varepsilon _k^{-\frac{1}{s}}, \end{aligned}$$
(1.5)
with some fixed \(\hat{C}>0\). In particular, one then has the most favorable scaling of the ranks with respect to \(\varepsilon _k\) that is possible when the singular value sequences are generic elements of \(\ell ^{p,\infty }\), and one obtains
$$\begin{aligned} ||\mathbf{u}- \mathbf{v}_k|| \le \hat{C}^s |\sigma |_{\ell ^{p,\infty }} \bigl ({{\mathrm{rank}}}(\mathbf{v}_k) \bigr )^{-s} \end{aligned}$$
as in (1.4).
In addition, we also consider the case of exponential-type decay \(\sigma _k \le C e^{-c k^\beta }\) for \(k\in {\mathbb {N}}\), where \(C,c,\beta >0\). In this case,
$$\begin{aligned} \left( \sum _{k>n} |\sigma _k|^2 \right) ^{\frac{1}{2}} \lesssim n^{1-\beta } e^{-c n^\beta } \lesssim e^{-\tilde{c} n^\beta } , \end{aligned}$$
for any \(\tilde{c} < c\) when \(\beta <1\), and with \(\tilde{c} = c\) when \(\beta \ge 1\). We thus obtain \(n_\varepsilon \lesssim ( 1 + |\ln \varepsilon |)^{\frac{1}{\beta }}\), and thus, quasi-optimal bounds are of the form
$$\begin{aligned} ||\mathbf{u}- \mathbf{v}_k|| \le \varepsilon _k , \qquad {{\mathrm{rank}}}(\mathbf{v}_k) \lesssim (1 + |\ln \varepsilon _k|)^{\frac{1}{\beta }}. \end{aligned}$$
(1.6)
In this case, rewriting the error bounds in terms of the ranks gives
$$\begin{aligned} ||\mathbf{u}- \mathbf{v}_k || \le \bar{C} \exp \Bigl (-\bar{c}\, \bigl ({{\mathrm{rank}}}(\mathbf{v})\bigr )^\beta \Bigr ) \end{aligned}$$
with \(\bar{C}, \bar{c}>0\). In contrast to the case of algebraically decaying singular values, quasi-optimal bounds on the ranks here mean that the exponent \(\beta \) is preserved in the corresponding error decay in terms of the ranks, with a modified prefactor \(\bar{c}\).

We establish estimates of the form (1.5) and (1.6) for the iterates of (1.3) with appropriately chosen \(\alpha _k\). In the case \(d>2\), these estimates are for the hierarchical tensor format. The ranks in the above estimates are then replaced by the maximum entry of the hierarchical rank tuple, and in the bounds, we obtain additional factors that are fixed powers of d.

Although our resulting procedure of the form (1.3) can in principle be formulated on infinite-dimensional Hilbert spaces, in this work we restrict our considerations concerning a numerically realizable version to fixed discretizations. In other words, in the form given here, the scheme applies either to infinite-dimensional \({\mathcal {H}}_i \simeq \ell ^2({\mathbb {N}})\) (which is of course not implementable in practice), or to a fixed finite-dimensional choice \({\mathcal {H}}_i \simeq {\mathbb {R}}^{n_i}\). That the iteration remains applicable on infinite-dimensional \({\mathcal {H}}_i\) is crucial, since this means that the method in itself does not introduce any dependence on finite tensor mode sizes \(n_i\). When suitable preconditioning is used such that the condition number of \({\mathcal {A}}\) remains bounded independently of \(n_i\) as well, the method is therefore robust with respect to the underlying discretization.

A further important point in this regard is that we do not make any restrictive assumptions on a low-rank structure of \({\mathcal {A}}\). For instance, one can reformulate problems posed on standard Sobolev spaces (e.g., on \(H^1\) on a product domain) in the present framework using suitable Riesz bases. This leads to \({\mathcal {A}}\) bounded on the tensor product space \(\mathcal {H}=\ell ^2({\mathbb {N}}^d)\) of the form (1.2). However, such \({\mathcal {A}}\) is in general not of bounded rank, and when one passes to a finite-dimensional setting, its ranks depend on the discretization; these issues are addressed in detail in [3]. The results of the present work apply also in such cases.

After describing the construction of the operation \(\mathbf {S}_{\alpha }\) in (1.3), we first identify choices of geometrically decreasing \(\alpha _k\) that lead to the desired rank estimates, provided that the asymptotic decay behavior of the hierarchical singular values of \(\mathbf{u}^*\) is known. We then construct a scheme which achieves the same type of rank bounds without using such knowledge. Based on an a posteriori criterion, the method adjusts \(\alpha _k\) to the unknown decay of the hierarchical singular values such that the quasi-optimal ranks are preserved. This method requires no a priori information beyond bounds on the spectrum of \({\mathcal {A}}\) and on the norm of \({\mathbf {f}}\). In a third step, we develop a perturbed version of the scheme that permits inexactly evaluated residuals.

1.2 Relation to Previous Work

The combination of a convergent iterative scheme with a low-rank truncation, such as the HSVD and its variants, has been proposed for solving problems in low-rank formats in a number of works, for instance in [4, 6, 7, 29, 33, 35, 37]. A typical problem is that the truncation is either done up to error tolerances that still ensure convergence of the iteration, in which case it is not clear how large the tensor ranks of iterates can be, or one always truncates up to predefined target rank, where one instead has the problem that convergence is only guaranteed under very restrictive conditions on the underlying iteration. In [7], for instance, it is demonstrated that convergence is preserved with fixed-rank truncations in solving (1.1), provided that \({\mathcal {A}}\) has condition number very close to one.

In greedy methods [10, 20], low-rank approximations are improved by iteratively adding terms without performing any rank truncation operations. The convergence of such methods can be proven under quite general assumptions, but the resulting ranks are typically far from optimal when \(d>2\).

A further common approach is based on optimizing component tensors in a tensor decomposition to minimize a suitable objective functional. Examples are methods based on the alternating least squares approach and the density matrix renormalization group [31, 51], as well as Riemannian optimization on fixed-rank tensor manifolds [13, 36]. For further details and references concerning such methods, see also [28, §10] and [25]. With these strategies based on local optimization, one can easily restrict the tensor ranks; in fact, for many such methods, the ranks need to be fixed a priori. However, for arbitrary starting values, convergence to approximate solutions is not ensured. Such methods can indeed fail to produce good approximations, for instance due to convergence to local minima of the objective functional.

The AMEn scheme [18] is a hybrid approach in that it uses residual information, combined with optimization strategies that exploit the structure of the tensor format. Here, convergence can in fact be shown. Although the scheme has been demonstrated to perform well on typical model problems, it needs to be noted that the convergence theory given in [18] does not apply to the practically realized method (which uses additional tensor truncations), and the rank bounds that one obtains from the theory scale exponentially in d and do not relate to the approximability of the solution.

For the case that the rank reduction in an iteration for solving (1.1) is done by a truncated HSVD (i.e., by hard thresholding), a scheme for choosing thresholding parameters that lead to near-optimal ranks is given in [2, 3]. To the authors’ knowledge, this is the only previous instance of a method that, under realistic requirements on \({\mathcal {A}}\), guarantees global converge to the true solution while at the same time, the arising ranks can be estimated in terms of the ranks required for approximating the solution. A limitation of the approach used there to control the ranks is that their near-optimality is enforced by truncating with a sufficiently large error tolerance, which can be done only after every few iterations when a certain error reduction has been achieved. The ranks of intermediate iterates can therefore still accumulate in the iterations between these complexity reductions and thus depend on the low-rank structure of the operator. At least concerning complexity bounds, this can be problematic if each application of the operator already causes a large rank increase (although this can be mitigated in practice, see Sect. 6). In the method proposed in this work, such an accumulation can be ruled out, and intermittent, sufficiently large increases in approximation errors that restore quasi-optimality are not required. A detailed comparison to the results of [2, 3] is given in Sect. 5.2.

Note that here we do not address the aspect of an adaptive underlying discretization of the problem as considered in [2, 3]. The version of our algorithm allowing inexact evaluation of residuals can, however, serve as a starting point for combining the method with adaptivity for identifying suitable discretizations in the course of the iteration. Furthermore, we expect that the concepts put forward here can also be used in the construction of adaptive methods for sparse basis representations.

Iterations using soft thresholding of sequences have been studied extensively in the context of inverse and ill-posed problems, see, e.g., [5, 8, 16], where they are especially well suited for obtaining convergence under very general conditions. Note that in such a setting, a priori choices of geometrically decreasing thresholding parameters have been proposed, e.g., in [14, 52]. Although the choice of thresholding parameters is an important issue also for ill-posed problems, they require entirely different strategies from the ones considered here, aiming at the identification of positive thresholding parameters that yield the desired regularization. Our approach for controlling the complexity of iterates—in the present case, the arising tensor ranks—in iterative schemes for well-posed problems, where the thresholding parameters converge to zero, appears to be new, in particular the a posteriori criterion that steers their decrease.

Soft thresholding of matrices, to which our procedure for hierarchical tensors reduces for \(d=2\), is also used in iterative methods for nuclear norm minimization in matrix completion, see[9, 42, 48], which are formally similar to (1.3), but have quite different features since the involved operators are far from being isomorphisms. In such a setting, dual gradient descent methods [40] can offer advantageous convergence behavior, but these exploit that iterates for the dual variable are in a relatively small linear space for completion problems and are thus not appropriate in our situation. In tensor completion, as a replacement of the nuclear norm in the matrix case, the sum of nuclear norms of matricizations has been proposed [32, 41]; however, with this approach, one does not recover the particular properties of the matrix case.1 In [21], repeated soft thresholding of matricizations of Tucker tensors is used in a splitting scheme for minimizing such sums of nuclear norms. This procedure bears some resemblance to our soft thresholding of hierarchical tensors, but eventually solves a different problem.

The method we propose here can also be motivated by a variational formulation of the operator equation. For instance, if \({\mathcal {A}}\) is symmetric, the solution is characterized by
$$\begin{aligned} \mathbf {u}^* = \mathop {\hbox {argmin}}\limits _{\mathbf {v} \in {\mathcal {H}}} \left\{ \frac{1}{2} \langle {\mathcal {A}} \mathbf {v} , \mathbf{v}\rangle - \langle {\mathbf {f}}, \mathbf{v}\rangle \right\} . \end{aligned}$$
A standard way to solve this problem in the spirit of Ritz–Galerkin methods would be to restrict it to the manifold of hierarchical tensors with fixed rank, which leads to the minimization-based approaches mentioned above. However, the sets over which one needs to minimize are not convex, and there generally exist many local minima. Roughly speaking, in such methods one fixes the model class (in this case by the admissible hierarchical tensor ranks) and attempts to minimize the error over this class. In this work, we aim to circumvent the difficulties caused by the nonlinearity of the representation in this setting, which manifest themselves in the problems with local minima. We thus let the ranks vary in the solution process and need to ensure that they remain of appropriate size.

In an alternative variational formulation, one can prescribe an error tolerance, for instance \(\Vert {\mathcal {A}}\mathbf{v}-{\mathbf {f}}\Vert \le \varepsilon \), and attempt to minimize the tensor ranks over the set of such \(\mathbf{v}\). Although the admissible set is then convex, even in the matrix case \(d=2\), the rank does not define a convex functional. However, one can instead minimize an appropriate convex relaxation, such as the \(\ell ^1\)-norm of singular values. It is well known that in the matrix case, such relaxed problems can be solved by proximal gradient methods, which can be rewritten as iterative soft thresholding [42] and hence take precisely the form (1.3) when \(d=2\). In this case, our method can therefore also be motivated as a rank minimization scheme, although this connection does not play a role in the analysis. Note, however, that in the case of higher-order tensors, where our soft thresholding procedure no longer permits an interpretation of the resulting scheme as a proximal gradient method, this is only a formal analogy.

1.3 Outline

This article is arranged as follows: In Sect. 2, we collect some prerequisites concerning the hierarchical tensor format as well as soft thresholding of sequences and of Hilbert–Schmidt operators. In Sect. 3, we then describe and analyze the new soft thresholding procedure for hierarchical tensors. In Sect. 4, we consider the combination of this procedure with general contractive fixed-point iterations and derive rank estimates for sequences of thresholding parameters that are chosen based on a priori information on the tensor approximability of \(\mathbf{u}^*\). In Sect. 5, we introduce an algorithm that automatically determines a suitable choice of thresholding parameters without using information on \(\mathbf{u}^*\), analyze its convergence, and additionally give a modified version of the scheme based on inexact residual evaluations. In Sect. 6, we conclude with numerical experiments that illustrate the practical performance of the proposed method.

2 Preliminaries

By \(||\cdot ||\), we always denote either the canonical norm on \(\mathcal {H}\), which is the product of the norms on the \(\mathcal {H}_i\), or the \(\ell ^2\)-norm when applied to a sequence.

For separable Hilbert spaces \(\mathcal {S}_1, \mathcal {S}_2\), we write \({\mathrm {HS}}(\mathcal {S}_1, \mathcal {S}_2)\) for the space of Hilbert–Schmidt operators from \(\mathcal {S}_1\) to \(\mathcal {S}_2\) with the Hilbert–Schmidt norm \(||\cdot ||_{\mathrm {HS}}\), which reduces to the Frobenius norm in the case of finite-dimensional spaces. Hilbert–Schmidt operators have a singular value decomposition (SVD) with singular values in \(\ell ^2\) satisfying the following perturbation estimate, shown for matrices in [44] and for Hilbert–Schmidt operators in [43].

Theorem 2.1

(cf. [43, Cor. 5.3]) Let \(\mathcal {S}_1, \mathcal {S}_2\) be separable Hilbert spaces, let \(\mathbf {X}, {\tilde{\mathbf{X}}} \in {\mathrm {HS}}(\mathcal {S}_1, \mathcal {S}_2)\), and let \(\sigma , \tilde{\sigma }\in \ell ^2({\mathbb {N}})\) denote the corresponding sequences of singular values. Then, \(|| \sigma - \tilde{\sigma }||_{\ell ^2({\mathbb {N}})} \le || \mathbf {X} - {\tilde{\mathbf{X}}} ||_{{\mathrm {HS}}}\).

2.1 The Hierarchical Tensor Format

We now briefly recall definitions and facts concerning the hierarchical Tucker format [30] and collect some basic observations that will play a role later. For further details on the hierarchical format, we refer to [28] and to the exhaustive treatment in [27].

Throughout this work, we assume \(d \ge 2\). For what follows, let \(\mathbb {T}\) be a binary dimension tree for tensor order d, that is, \(\mathbb {T}\) is a set of non-empty subsets of \(\{1,\ldots ,d\}\) such that \(\{1,\ldots ,d\} \in \mathbb {T}\) and for each \(n\in \mathbb {T}\), either \(\#n=1\) or there exist disjoint \(n_1,n_2\in \mathbb {T}\) such that \(n_1\cup n_2 = n\). Examples of such dimension trees for \(d=4\) are given in Fig. 1. A choice of \(\mathbb {T}\) that is closely related to the tensor train format is the linear dimension tree
$$\begin{aligned} \mathbb {T} = \bigl \{ \{1,\ldots ,d\} , \{1\}, \{2,\ldots ,d\}, \{2\}, \{ 3,\ldots , d\}, \ldots , \{d-1\}, \{d\} \bigr \}. \end{aligned}$$
(2.1)
Fig. 1

Examples of equivalent dimension trees obtained by moving the root element

For each node \(n\in \mathbb {T}\), we set \(n^c{:}{=} \{ 1,\ldots , d\}{\setminus } n\). Note that in general, \(n^c\notin \mathbb {T}\). For \(n\in \mathbb {T}\), we define the matricization\({\mathcal {\hat{M}}}_n(\mathbf{u})\) as the canonical reinterpretation of \(\mathbf{u}\in \bigotimes _{i =1}^d \mathcal {H}_i\) as a Hilbert–Schmidt operator
$$\begin{aligned} {\mathcal {\hat{M}}}_n(\mathbf{u}) :\bigotimes _{i \in n^c} \mathcal {H}_i \rightarrow \bigotimes _{i \in n} \mathcal {H}_i. \end{aligned}$$
For \(n,n_1,n_2\in \mathbb {T}\) with \(n=n_1\cup n_2\) and \( {{\mathcal {U}}}_n = {{\mathcal {U}}}_n(\mathbf{u}) {:}{=} \overline{ {\text {range}} {\mathcal {\hat{M}}}_n(\mathbf{u}) } \), one has the crucial nestedness property
$$\begin{aligned} {\mathcal {U}}_n \subseteq {\mathcal {U}}_{n_1} \otimes {\mathcal {U}}_{n_2}. \end{aligned}$$
(2.2)
We now introduce the set of effective edges
$$\begin{aligned} \mathbb {E} {:}{=} \bigl \{ \{ n , n^c\} :n\in \mathbb {T}{\setminus } \{1,\ldots ,d\} \bigr \}. \end{aligned}$$
The elements of \(\mathbb {E}\) correspond to the edges in the tree \(\mathbb {T}\) as in Fig. 1, except that the two lines connected to the root element \( \{1,\ldots ,d\}\) form a single effective edge. We set
$$\begin{aligned} E{:}{=} \#\mathbb {E} = 2 d - 3. \end{aligned}$$
For what follows, we always assume a fixed enumeration \(\bigl (\{ n_t, n_t^c \} \bigr )_{t=1,\ldots , E}\) of \(\mathbb {E}\), where \(n_t\in \mathbb {T}\) for \(t=1,\ldots ,E\). Note that the efficiency of the soft thresholding operation for hierarchical tensors that we will describe depends on this sequence; this point will be considered in further detail in Sect. 3. For each \(n\in \mathbb {T}{\setminus }\{1,\ldots ,d\}\), we denote by t(n) the uniquely determined element of \(\{1,\ldots ,E\}\) such that \(n=n_{t(n)}\) or \(n=n_{t(n)}^c\).
For \(t=1,\ldots ,E\) with \(\{n_t, n_t^c\} \in \mathbb {E}\), we define by \(\mathcal {M}_t(\mathbf{u}) {:}{=} {\mathcal {\hat{M}}}_{n_t}(\mathbf{u})\) the t-matricization of the tensor \(\mathbf{u}\). By \(\mathcal {M}^{-1}_t\), we denote the mapping that converts this matricization back to a tensor. Note that for each t, one has \(\mathcal {M}_t(\mathbf{u}) + \mathcal {M}_t(\mathbf{v}) = \mathcal {M}_t(\mathbf{u}+\mathbf{v})\) and
$$\begin{aligned} ||\mathbf{u}|| = ||\mathcal {M}_t(\mathbf{u})||_{\mathrm {HS}}. \end{aligned}$$
(2.3)
The sequence of singular values of this matricization is denoted by \(\sigma _t (\mathbf{u})\), which we always assume to be defined on \({\mathbb {N}}\) (with extension by zero in the case of finite-dimensional \(\mathcal {H}_i\)), and we set
$$\begin{aligned} {{\mathrm{rank}}}_t(\mathbf{u}) {:}{=} {{\mathrm{rank}}}\bigl (\mathcal {M}_t(\mathbf{u})\bigr ) = \# \{ k\in {\mathbb {N}}:\sigma _{t,k}(\mathbf{u}) \ne 0 \} \;\in \; {\mathbb {N}}_0 \cup \{ \infty \}. \end{aligned}$$
When the tensor \(\mathbf{u}\) under consideration is fixed, we also use the abbreviation \(r(n){:}{=} {{\mathrm{rank}}}_{t(n)}(\mathbf{u})\) in the remainder of this section.
As a consequence of (2.2), choosing bases for the \({\mathcal {U}}_n(\mathbf{u})\) yields a recursive decomposition of \(\mathbf{u}\) as follows. For each \(n\in \mathbb {T}{\setminus }\{1,\ldots ,d\}\), let \(\{ {\mathbf {U}}^{(n)}_k \}_{k=1,\ldots ,r(n)}\) be an arbitrary orthonormal basis of \({\mathcal {U}}_n\). As a consequence of (2.2), for \(n \ne \{1,\ldots ,d\}\) with \(n=n_1\cup n_2\),
$$\begin{aligned} {\mathbf {U}}^{(n)}_k = \sum _{k_1 = 1}^{r(n_1)} \sum _{k_2 = 1}^{r(n_2)} \mathbf {B}^{(n)}_{k,k_1,k_2} {\mathbf {U}}^{(n_1)}_{k_1} \otimes {\mathbf {U}}^{(n_2)}_{k_2} , \quad k = 1,\ldots , r(n), \end{aligned}$$
(2.4)
where \(\mathbf {B}^{(n)}_{k,k_1,k_2} {:}{=} \langle {\mathbf {U}}^{(n)}_k, {\mathbf {U}}^{(n_1)}_{k_1} \otimes {\mathbf {U}}^{(n_2)}_{k_2} \rangle \), and
$$\begin{aligned} \mathbf{u}= \sum _{k_1 = 1}^{r(n_1)} \sum _{k_2 = 1}^{r(n_2)} \mathbf {C}_{k_1,k_2} {\mathbf {U}}^{(n_1)}_{k_1} \otimes {\mathbf {U}}^{(n_2)}_{k_2} \end{aligned}$$
(2.5)
when \(\{1,\ldots ,d\}=n_1\cup n_2\), where one has \(r(n_1)=r(n_2)\).
We adopt the terminology of [24], referring to the collections of basis vectors \(\{ {\mathbf {U}}^{(\{i\})}_k \}, i=1,\ldots ,d\), in the leaves of \(\mathbb {T}\) as mode frames, and to the coefficient tensors \(\mathbf {B}^{(n)}\) and \(\mathbf {C}\) at interior nodes as transfer tensors. Combining (2.5) and (2.4) recursively, one thus obtains a representation of \(\mathbf{u}\) only in terms of transfer tensors and mode frames, that is, in terms of tensors of order up to three. When such a representation uses an orthonormal basis for each \({\mathcal {U}}_n\) as we have supposed here, we refer to it as an orthonormal hierarchical representation. For such a representation, an SVD \(\mathbf {C}_{ij} = \sum _{\ell =1}^{r(n_1)} \sigma _\ell \mathbf {V}^{(n_1)}_{i,\ell } \mathbf {V}^{(n_2)}_{j,\ell } \) of \(\mathbf {C}\) as in (2.5) yields the SVD
$$\begin{aligned} \mathcal {M}_t(\mathbf{u}) = \sum _{\ell =1}^{r(n_1)} \sigma _\ell \left( \sum _{k_1} {\mathbf {U}}^{(n_1)}_{k_1} \mathbf {V}^{(n_1)}_{k_1 , \ell } \right) \left( \sum _{k_2} {\mathbf {U}}^{(n_2)}_{k_2} \mathbf {V}^{(n_2)}_{k_2 , \ell } \right) ^*, \end{aligned}$$
(2.6)
where \(t = t(n_1)=t(n_2)\) corresponds to the edge holding the root of \(\mathbb {T}\). Using this observation recursively leads to the HSVD [24]; for our present purposes, however, we shall only need (2.6).

The effective edges \(\mathbb {E}\) give rise to an equivalence relation between different dimension trees for a given d: In general, there are several dimension trees \(\mathbb {T}\) that share the same corresponding \(\mathbb {E}\), that is, involve the same matricizations of the tensor. The difference between \(\mathbb {T}\) that are equivalent in this sense is that they have the root element of the tree at a different effective edge. This is illustrated in Fig. 1 for a tensor of order four.

Moving the root element in the tensor representation can be done in practice by basic linear algebra manipulations, where the transfer tensors in the representation are relabelled and reorthogonalized accordingly. This can be formulated generically as follows: Let \(\mathbf{u}\) be given in orthonormal hierarchical representation, let \(n\in \mathbb {T}\) be a son of \(\{1,\ldots ,d\}\), and let \(n_1,n_2\) be the sons of n, so that
$$\begin{aligned} \mathbf{u}= \sum _{k_1 , k_2 = 1}^{r(n)} \sum _{\ell _1=1}^{r(n_1)} \sum _{\ell _2=1}^{r(n_2)} \mathbf {C}_{k_1,k_2} \mathbf {B}^{(n)}_{k_1,\ell _1,\ell _2} {\mathbf {U}}^{(n_1)}_{\ell _1} \otimes {\mathbf {U}}^{(n_2)}_{\ell _2} \otimes {\mathbf {U}}^{(n^c)}_{k_2} , \end{aligned}$$
(2.7)
where \(r(n)=r(n^c)\). Here, the \({\mathbf {U}}^{(n_1)}_{\ell _1}, {\mathbf {U}}^{(n_2)}_{\ell _2}, {\mathbf {U}}^{(n^c)}_{k_2}\) are in turn expanded in terms of transfer tensors and mode frames; note that we assume without loss of generality that the sets \(n_1,n_2,n^c\) correspond to an ascending ordering of dimensions. We would now like to change the representation so that the root element is at the effective edge \(\{ n_1, n_1^c\}\).
To this end, we build the matrix \(\mathbf {T}\in {\mathbb {R}}^{(r(n_2)\,r(n))\times r(n_1)}\) with entries
$$\begin{aligned} \mathbf {T}_{(\ell _2,k_2), \ell _1} {:}{=} \sum _{k_1=1}^{r(n)} \mathbf {C}_{k_1,k_2} \mathbf {B}^{(n)}_{k_1,\ell _1,\ell _2} \end{aligned}$$
and factorize it in the form \(\mathbf {T} = \mathbf {Q}\mathbf {R}\), where \(\mathbf {Q}\in {\mathbb {R}}^{(r(n_2)\,r(n))\times r(n_1)}\) has orthonormal columns. We set \({\tilde{\mathbf{C}}} = \mathbf {R}^T\) and \({\tilde{\mathbf{B}}}^{(n_1^c)}_{\tilde{\ell }, \ell _2, k_2} {:}{=} \mathbf {Q}_{(\ell _2,k_2), \tilde{\ell }}\) to obtain
$$\begin{aligned} \mathbf{u}= \sum _{\ell _1 , \tilde{\ell }= 1}^{r(n_1)} \sum _{\ell _2=1}^{r(n_2)} \sum _{k_2=1}^{r(n^c)} {\tilde{\mathbf{C}}}_{\ell _1,\tilde{\ell }} {\tilde{\mathbf{B}}}^{(n_1^c)}_{\tilde{\ell },\ell _2,k_2} {\mathbf {U}}^{(n_1)}_{\ell _1} \otimes {\mathbf {U}}^{(n_2)}_{\ell _2} \otimes {\mathbf {U}}^{(n^c)}_{k_2}. \end{aligned}$$
(2.8)
This orthonormal hierarchical representation has the same effective edges \(\mathbb {E}\), but a different dimension tree, where \(n_1\) and \(n_1^c=n_2\cup n^c\) are now the sons of the root element \(\{1,\ldots ,d\}\). By an appropriate choice of \(n,n_1,n_2\), this procedure can be used for moving the root element from any edge to an adjacent one.

2.2 Soft Thresholding

For \(x \in {\mathbb {R}}\), soft thresholding with parameter \(\alpha > 0\) is defined by
$$\begin{aligned} s_{\alpha } (x) {:}{=} {{\mathrm{sgn}}}(x) \max \{ |x| - \alpha , 0 \}. \end{aligned}$$
(2.9)
In comparison, hard thresholding is given by Open image in new window.

Applied to each element of a vector or sequence, hard thresholding provides a very natural means of obtaining sparse approximations by dropping entries of small absolute value, which is closely related to best n-term approximation [17]. Soft thresholding not only replaces entries that have absolute value below the threshold by zero, but also decreases all remaining entries, incurring an additional error. However, this operation has a non-expansiveness property that is useful in the construction of iterative schemes. This property can be derived from a variational characterization.

To describe this characterization, for a proper, closed convex functional \({\mathcal {J}} : {\mathcal {G}} \rightarrow \mathbb {R} \) on a Hilbert space \({\mathcal {G}}\) and \( \alpha \ge 0\), following [45], we define the proximity operator\({{\mathrm{prox}}}^{\alpha }_{{\mathcal {J}}} : {\mathcal {G}} \rightarrow {\mathcal {G}}\) by
$$\begin{aligned} {{\mathrm{prox}}}^{\alpha }_{{\mathcal {J}}} (\mathbf {u}) {:}{=} \mathop {\hbox {argmin}}\limits _{\mathbf {v} \in {\mathcal {G}} } \Bigl \{ \alpha {\mathcal {J}}(\mathbf {v}) + \frac{1}{2 } || \mathbf {u} - \mathbf {v} ||^2_{\mathcal {G}} \Bigr \} \ . \end{aligned}$$
As shown in [45], such operators have the following general property, which plays a crucial role in this work.

Lemma 2.2

The proximity operator \( {{\mathrm{prox}}}^{\alpha }_{{\mathcal {J}}} \) is non-expansive, that is,
$$\begin{aligned} || {{\mathrm{prox}}}^{\alpha }_{{\mathcal {J}}} (\mathbf {u} ) - {{\mathrm{prox}}}^{\alpha }_{{\mathcal {J}}} (\mathbf {v} ) ||_{\mathcal {G}} \le || \mathbf {u}-\mathbf {v}||_{\mathcal {G}},\quad \mathbf{u},\mathbf{v}\in {\mathcal {G}}. \end{aligned}$$
The scalar soft thresholding operation defined in (2.9) can be characterized as the proximity operator of the absolute value, that is,
$$\begin{aligned} s_\alpha (x) = \mathop {\hbox {argmin}}\limits _{y\in {\mathbb {R}}} \Bigl \{ \frac{1}{2} |x-y|^2 + \alpha |y| \Bigr \} ,\quad x \in {\mathbb {R}}. \end{aligned}$$
(2.10)
Soft thresholding of a sequence \(\mathbf{v}\) can be defined by applying \(s_\alpha \) to each entry, \(s_\alpha (\mathbf{v}) {:}{=} \bigl (s_\alpha (v_i)\bigr )_{i\in {\mathbb {N}}}\), and can be characterized as the proximity operator of the functional \(||\cdot ||_{\ell ^1}\) (cf. [16]). In other words,
$$\begin{aligned} s_\alpha (\mathbf{v}) = \mathop {\hbox {argmin}}\limits _{\mathbf{w}\in \ell ^2} \Bigl \{ \frac{1}{2} ||\mathbf{v}- \mathbf{w}||_{\ell ^2}^2 + \alpha ||\mathbf{w}||_{\ell ^1} \Bigr \} , \end{aligned}$$
(2.11)
where the minimum on the right is obtained componentwise by (2.10), and \(s_\alpha \) is therefore a non-expansive mapping on \(\ell ^2\).
An analogous characterization is still possible when soft thresholding is applied to the singular values of matrices or operators, which provides a reduction to lower matrix ranks. More precisely, the soft thresholding operation \(S_\alpha \) for matrices is defined as follows: For a given matrix \(\mathbf {X}\) with singular value decomposition
$$\begin{aligned} \mathbf {X} = {\mathbf {U}} {{\mathrm{diag}}}\bigl ( {\sigma }_i (\mathbf {X})\bigr ) \mathbf {V}^T, \end{aligned}$$
where \({\sigma }_i (\mathbf {X})\) denotes the i-th singular value of \(\mathbf {X}\), we set
$$\begin{aligned} S_{\alpha } (\mathbf {X}):= {\mathbf {U}} {{\mathrm{diag}}}\bigl (s_{\alpha } ({\sigma }_i(\mathbf {X}))\bigr )\mathbf {V}^T. \end{aligned}$$
Note that application of the hard thresholding \(h_\alpha \) to the singular values would instead correspond to a rank truncation of the singular value decomposition. For Hilbert–Schmidt operators \(\mathbf {X}\), one can define \(S_\alpha (\mathbf {X})\) analogously.
As a consequence of the following result, the mapping \(S_\alpha \) is the proximity operator corresponding to the nuclear norm,
$$\begin{aligned} ||\mathbf {X}||_* :=||\sigma (\mathbf {X})||_{\ell ^1} = \sum _{i \ge 1} \sigma _i(\mathbf {X}). \end{aligned}$$

Lemma 2.3

Let \(\mathcal {S}_1, \mathcal {S}_2\) be separable Hilbert spaces, \(\mathbf {X} \in {\mathrm {HS}}(\mathcal {S}_1, \mathcal {S}_2)\), and \(\alpha \ge 0\). Then,
$$\begin{aligned} S_{\alpha } (\mathbf {X}) = \mathop {\hbox {argmin}}\limits _{{\tilde{\mathbf{X}}} \in {\mathrm {HS}}} \Bigl \{ \alpha \Vert {\tilde{\mathbf{X}}}\Vert _{*} + \frac{1}{2} \Vert \mathbf {X} - {\tilde{\mathbf{X}}} \Vert ^2_{\mathrm {HS}}\Bigr \}, \end{aligned}$$
(2.12)
or in other words, \({{\mathrm{prox}}}^\alpha _{||\cdot ||_*} (\mathbf {X} ) = S_{\alpha } (\mathbf {X})\).

This statement is shown for finite matrices \(\mathbf {X}\), e.g., in [9] using subgradient characterizations. Based on Theorem 2.1, we can give the following argument for Hilbert–Schmidt operators.

Proof

Let \({\tilde{\mathbf{X}}}\in {\mathrm {HS}}\) such that the right-hand side in (2.12) is finite, which implies that \({\tilde{\mathbf{X}}}\) is trace class, and let \(\sigma , \tilde{\sigma }\) denote the sequences of singular values of \(\mathbf {X}, {\tilde{\mathbf{X}}}\), respectively. By (2.11) and Theorem 2.1, we thus have
$$\begin{aligned} \frac{1}{2} ||\mathbf {X} - S_\alpha (\mathbf {X}) ||_{\mathrm {HS}}^2 + \alpha ||S_\alpha (\mathbf {X})||_{*}= & {} \frac{1}{2} ||\sigma - s_\alpha (\sigma )||_{\ell ^2}^2 +\alpha ||s_\alpha (\sigma )||_{\ell ^1}\\\le & {} \frac{1}{2} ||\sigma - \tilde{\sigma }||_{\ell ^2}^2 + \alpha ||\tilde{\sigma }||_{\ell ^1}\\\le & {} \frac{1}{2} ||\mathbf {X} - {\tilde{\mathbf{X}}} ||_{\mathrm {HS}}^2 + \alpha ||{\tilde{\mathbf{X}}}||_{*}. \end{aligned}$$
Since the minimum is unique by strict convexity, we obtain the assertion. \(\square \)

3 Soft Thresholding of Hierarchical Tensors

In this section, we construct a non-expansive soft thresholding operation for the rank reduction in hierarchical tensors. By \(S_{t,{\alpha } }: {\mathcal {H}} \rightarrow {\mathcal {H}}\), we denote soft thresholding applied to the matricization \(\mathcal {M}_t(\cdot )\) of the input,
$$\begin{aligned} S_{t,\alpha } (\mathbf{u}) {:}{=} \bigl ( \mathcal {M}^{-1}_t \circ S_{\alpha } \circ \mathcal {M}_t \bigr )(\mathbf{u}). \end{aligned}$$
The complete hierarchical tensor soft shrinkage operator \({\mathbf {S}}_{{\alpha }} : \mathcal {H}\rightarrow \mathcal {H}\) is then defined as the successive application of this operation to each matricization, that is,
$$\begin{aligned} {\mathbf {S}}_{{\alpha } } (\mathbf {u} ) : = \bigl ( S_{E, \alpha } \circ \cdots \circ S_{1, \alpha } \bigr ) ( \mathbf {u} ) \ . \end{aligned}$$
(3.1)
Clearly, the result of applying \({\mathbf {S}}_\alpha \) with \(\alpha > 0\) always has finite (but not a priori fixed) hierarchical ranks.

For a hierarchical tensor \(\mathbf {u}\) with suitably numbered edges \(\{n_t,n_t^c\}\in \mathbb {E}, t = 1, \ldots , E\), the soft thresholding \( {\mathbf {S}}_{{\alpha }} (\mathbf {u}) \) can be obtained as follows: Starting with \(\mathbf{u}_0 = \mathbf{u}\), for each t, we first rearrange \(\mathbf{u}_{t-1}\) such that the root element is on \(\{n_t,n_t^c\}\) with a singular value decomposition of \(\mathcal {M}_t(\mathbf{u}_{t-1})\), which exposes the singular values \(\sigma _t(\mathbf{u}_{t-1})\) and thus allows the direct application of \(S_{t,\alpha }\) to obtain \(\mathbf{u}_t {:}{=} S_{t,\alpha }(\mathbf{u}_{t-1})\). This yields \(\mathbf {S}_\alpha (\mathbf{u}) = \mathbf{u}_E\).

Our further theory applies to any order of enumerating \(\mathbb {E}\), but this order does matter for the resulting number of operations. An example of an order in which the edges in \(\mathbb {E}\) can be visited in the case \(d=4\) is shown in Fig. 1. A corresponding general procedure can be described as follows:
  • Start with an orthonormal hierarchical representation of \(\mathbf{u}\) such that the root element is at \(\{n_1,n_1^c\}{:}{=}\{ \{1\},\{2,\ldots ,d\}\}\in \mathbb {E}\), perform an SVD of \(\mathcal {M}_1(\mathbf{u})\), and use this to obtain \(\mathbf{u}_1{:}{=} S_{1,\alpha }(\mathbf{u})\).

  • If \(d=2\), we are done; if \(d>2\), call \({{\mathrm{\textsc {STRecursion}}}}_\alpha (\mathbf{u}_1, \{2,\ldots ,d\})\) as defined in Algorithm 1.

Moving the root element of the dimension tree as required for this procedure amounts to performing the steps for going from (2.7) to (2.8) described in Sect. 2.1.

Remark 3.1

Assuming that we are given a hierarchical tensor with \(\dim {\mathcal {H}}_i \le n \in {\mathbb {N}}\) and representation ranks bounded by r, then the first step of bringing this tensor into its initial representation takes \({\mathcal {O}}(d r^4 + dr^2n)\) operations. Inspecting the procedure given in Sect. 2.1, one finds that moving the root element from one edge to an adjacent one costs \({\mathcal {O}}(r^4)\) operations, and handling the arising SVDs is of the same order. In the general procedure we have described, the root is moved across each interior node of the tree exactly three times (which is clearly also the minimum number possible), at total cost \({\mathcal {O}}(d r^4)\). The total complexity for applying \(\mathbf {S}_\alpha \) applied in this manner is thus of order \({\mathcal {O}}(d r^4 + dr^2n)\), which is the same as for computing the HSVD.

Proposition 3.2

For any \(\mathbf{u}, \mathbf{v}\in \mathcal {H}\) and \(\alpha >0\), the operator \({\mathbf {S}}_\alpha \) defined in (3.1) satisfies \(||{\mathbf {S}}_\alpha (\mathbf{u}) - {\mathbf {S}}_\alpha (\mathbf{v})|| \le ||\mathbf{u}- \mathbf{v}||\).

Proof

The statement follows by repeated application of Lemmata 2.2, 2.3 and (2.3). \(\square \)

The following lemma guarantees that applying soft thresholding to a certain matricization of a tensor does not increase the hierarchical singular values of any other matricization of this tensor.

Lemma 3.3

For any \(\mathbf{v}\in \mathcal {H}\) and for \(t, s = 1, \ldots , E\), one has \(\sigma _{t,i}( \mathbf{u}) \ge \sigma _{t,i}( S_{s,\alpha } (\mathbf{u}))\) for all \(i\in {\mathbb {N}}\) and any \(\alpha \ge 0\).

Proof

Note that for the action of \(S_{s,\alpha }\), the tensor is rearranged such that the edge s holds the root element. Thus, the statement follows exactly as in part 3 of the proof of Theorem 11.61 in [27], see also the proof of Theorem 7.18 in [39]; there it is shown that when singular values are decreased at the root element, this cannot cause any singular value of the other matricizations in the dimension tree to increase. \(\square \)

Using the above lemma, the error incurred by application of \(\mathbf {S}_\alpha \) to a tensor \(\mathbf{u}\) can be estimated in terms of the sequences of hierarchical singular values \(\sigma _t(\mathbf{u}), t=1,\ldots , E\).

Lemma 3.4

For any \(\mathbf{u}\in \mathcal {H}\) and \(r\in {\mathbb {N}}_0\), let
$$\begin{aligned} \tau _{t,r}(\mathbf{u}) {:}{=} \inf _{{{\mathrm{rank}}}_t(\mathbf{w}) \le r} ||\mathbf{u}- \mathbf{w}|| = \left( \sum _{ k > r } |\sigma _{t,k}(\mathbf{u})|^2\right) ^{\frac{1}{2}}. \end{aligned}$$
Furthermore, for any \(\delta >0\), we define
$$\begin{aligned} r_{t,\delta }(\mathbf{u}) {:}{=} \max \bigl ( \{ r\in {\mathbb {N}}:\sigma _{t,r} > \delta \} \cup \{ 0 \} \bigr ). \end{aligned}$$
Then, for any given \(\alpha >0\),
$$\begin{aligned} \max _{t=1,\ldots ,E} d^\alpha _t(\mathbf{u}) \le || {\mathbf {S}}_\alpha (\mathbf{u}) - \mathbf{u}|| \le \sum _{t=1}^E d^\alpha _t(\mathbf{u}) \le E \max _{t=1,\ldots ,E} d^\alpha _t(\mathbf{u}), \end{aligned}$$
(3.2)
where, with \(\tau _{t,\alpha }(\mathbf{u}){:}{=}\tau _{t,r_{t,\alpha }(\mathbf{u})}(\mathbf{u})\),
$$\begin{aligned} d_t^\alpha (\mathbf{u}) {:}{=} \bigl || \sigma _t \bigl (S_{t,\alpha }(\mathbf{u}) \bigr ) - \sigma _t(\mathbf{u}) \bigr || = \sqrt{ \alpha ^2 r_{t,\alpha }(\mathbf{u}) + \bigl (\tau _{t,\alpha }(\mathbf{u})\bigr )^2}. \end{aligned}$$

Remark 3.5

It can be seen as follows that the second inequality in (3.2) is sharp in certain cases: Choose \(\mathbf{u}\) as a tensor of rank one (i.e., with all hierarchical ranks equal to one) with \(||\mathbf{u}|| = E\alpha \), then \({\mathbf {S}}_\alpha (\mathbf{u}) = 0\). Since here one has \(r_{t,\alpha }(\mathbf{u})=1\) and \(\tau _{t,\alpha }(\mathbf{u})=0\) for all t and \(\alpha \), this gives
$$\begin{aligned} || {\mathbf {S}}_\alpha (\mathbf{u}) - \mathbf{u}|| = ||\mathbf{u}|| = E\alpha = \sum _{t=1}^E d^\alpha _t(\mathbf{u}) . \end{aligned}$$
In particular, the upper bound in terms of \(\max _{t=1,\ldots ,E} d^\alpha _t(\mathbf{u})\) in (3.2) thus cannot be improved as far as the d-dependence of the arising constant \(E\sim d\) is concerned.

Proof of Lemma 3.4

We first show the second inequality in (3.2). Let
$$\begin{aligned} \mathbf{v}_1 {:}{=} \mathbf{u},\qquad \mathbf{v}_t {:}{=} S_{t-1,\alpha } \circ \cdots \circ S_{1, \alpha }(\mathbf{u}) , \quad t \ge 2. \end{aligned}$$
By a telescoping sum argument, applying the soft thresholding error estimate to each individual application of \(S_{t,\alpha }\), we obtain
$$\begin{aligned} || {\mathbf {S}}_\alpha (\mathbf{u}) - \mathbf{u}|| \le \sum _{t=1}^E ||S_{t,\alpha }(\mathbf{v}_t) - \mathbf{v}_t|| \le \sum _{t=1}^E d^\alpha _t (\mathbf{v}_t). \end{aligned}$$
It remains to show that \( d^\alpha _t(\mathbf{v}_t ) \le d^\alpha _t(\mathbf{u}) \). This follows from Lemma 3.3, whose repeated application gives \(\sigma _{t,i}( \mathbf{u}) \ge \sigma _{t,i}( S_{1,\alpha }(\mathbf{u})) \ge \sigma _{t,i}(S_{2,\alpha } \circ S_{1,\alpha }(\mathbf{u})) \ge \ldots \) for each t.
To show the first inequality in (3.2), we again invoke Lemma 3.3, in this case to infer that for each t,
$$\begin{aligned} \sum _{i\ge 1} |\sigma _{t,i}(\mathbf {S}_\alpha (\mathbf{u})) - \sigma _{t,i}(\mathbf{u}) |^2 \ge \sum _{i\ge 1} |\sigma _{t,i}(S_{t,\alpha }(\mathbf{u})) - \sigma _{t,i}(\mathbf{u}) |^2 = \bigl ( d_t^\alpha (\mathbf{u}) \bigr )^2. \end{aligned}$$
Moreover, by Theorem 2.1, we have
$$\begin{aligned} ||\sigma _{t}(\mathbf {S}_\alpha (\mathbf{u})) - \sigma _{t}(\mathbf{u})|| \le ||\mathcal {M}_t(\mathbf {S}_\alpha (\mathbf{u})) - \mathcal {M}_t(\mathbf{u})||_{\mathrm {HS}}= ||\mathbf {S}_\alpha (\mathbf{u}) - \mathbf{u}||, \end{aligned}$$
and taking the maximum over t concludes the proof. \(\square \)

Let \(\mathbf{u}\in \mathcal {H}\), then \( \sigma _t(\mathbf{u}) \in \ell ^2\), which implies that \(d^\alpha _t(\mathbf{u}) \rightarrow 0\) as \(\alpha \rightarrow 0\). Without further assumptions, however, this convergence can be arbitrarily slow. In the following proposition, we collect results that quantify this convergence in terms of the decay of the hierarchical singular values.

Proposition 3.6

  1. (i)
    If in addition \(\sigma _{t}(\mathbf{u}) \in \ell ^{p,\infty }, t=1,\ldots ,E\), for a \(p \in (0,2)\), one has
    $$\begin{aligned} r_{t,\alpha } \lesssim |\sigma _t(\mathbf{u})|_{\ell ^{p,\infty }}^p \, \alpha ^{-p},\quad \tau _{t,\alpha } \lesssim \, |\sigma _t(\mathbf{u})|^{p/2}_{\ell ^{p,\infty }} \, \alpha ^{1 - p /2 }, \end{aligned}$$
    (3.3)
    and thus
    $$\begin{aligned} || {\mathbf {S}}_\alpha (\mathbf{u}) - \mathbf{u}|| \lesssim E\, \max _{t=1,\ldots ,E} |\sigma _t(\mathbf{u})|^{p/2}_{\ell ^{p,\infty }} \, \alpha ^{1-p/2} , \end{aligned}$$
    (3.4)
    where the constants depend only on p.
     
  2. (ii)
    If \(\sigma _{t,j}(\mathbf{u}) \le C e^{- c j^\beta }\) for \(j\in {\mathbb {N}}\) and \(t=1,\ldots ,E\), with \(C,c , \beta > 0\), then
    $$\begin{aligned} r_{t,\alpha } \le \bigl ( c^{-1} \ln (C\alpha ^{-1}) \bigr )^{\frac{1}{\beta }} \lesssim (1 + |\ln \alpha |)^{\frac{1}{\beta }}, \quad \tau _{t,\alpha } \lesssim (1 + |\ln \alpha |)^{\frac{1}{2\beta }} \,\alpha , \end{aligned}$$
    and therefore
    $$\begin{aligned} || {\mathbf {S}}_\alpha (\mathbf{u}) - \mathbf{u}|| \lesssim E\, (1 + |\ln \alpha |)^{\frac{1}{2\beta }} \, \alpha , \end{aligned}$$
    (3.5)
    with constants that depend on Cc, and \(\beta \).
     

Proof

The estimates (3.3) are shown in [17], and (3.4) follows with (3.2). In the same manner, we obtain (3.5) by arguing analogously to [17, Section7.4]. \(\square \)

Remark 3.7

As mentioned in the introduction, soft thresholding is closely related to convex optimization by proximal operator techniques. Note that the soft thresholding for hierarchical tensors can be written as
$$\begin{aligned} {\mathbf {S}}_{\alpha } = S_{E, \alpha } \circ \cdots \circ S_{1, \alpha } = {{\mathrm{prox}}}^{\alpha }_{J_E } \circ \cdots \circ {{\mathrm{prox}}}^{\alpha }_{J_1 } \end{aligned}$$
with \(J_t {:}{=} ||\mathcal {M}_t( \cdot ) ||_*\) for \(t=1,\ldots ,E\). Thus, in our setting, we do not have a characterization of \(\mathbf {S}_\alpha \) by a single convex optimization problem (as provided for \(S_\alpha \) by Lemma 2.3), but still by a nested sequence of convex optimization problems: One has \(\mathbf {S}_\alpha (\mathbf{u}) = \tilde{\mathbf{u}}_E\), where
$$\begin{aligned} {\tilde{\mathbf{u}}}_t {:}{=} \mathop {\hbox {argmin}}\limits _{\mathbf{v}\in \mathcal {H}} \Bigl \{ J_t(\mathbf{v}) + \frac{1}{2\alpha } ||\tilde{\mathbf{u}}_{t-1} - \mathbf{v}||^2 \Bigr \}, \quad t = 1,\ldots , E, \end{aligned}$$
with \(\tilde{\mathbf{u}}_0 {:}{=} \mathbf{u}\).

4 Fixed-Point Iterations with Soft Thresholding

In this section, we consider the combination of \(\mathbf {S}_\alpha \) with an arbitrary convergent fixed-point iteration given by a contractive mapping \(\mathcal {F}:\mathcal {H}\rightarrow \mathcal {H}\), that is, by \(\mathcal {F}\) for which there exists \(\rho \in (0,1)\) such that
$$\begin{aligned} ||\mathcal {F}(\mathbf{v}) - \mathcal {F}(\mathbf{w})|| \le \rho ||\mathbf{v}- \mathbf{w}||,\quad \mathbf{v}, \mathbf{w}\in \mathcal {H}. \end{aligned}$$
(4.1)
In the example of a linear operator equation \({\mathcal {A}}\mathbf{u}= {\mathbf {f}}\) with elliptic and bounded \({\mathcal {A}}\), we may choose \(\mathcal {F}(\mathbf{v}) = \mathbf{v}- \omega ({\mathcal {A}}\mathbf{v}- {\mathbf {f}})\) with a suitable scaling parameter \(\omega >0\). A practical scheme for this particular case will be considered in detail in Sect. 5.

Since \(\mathbf {S}_\alpha \) is non-expansive, the mapping \(\mathbf {S}_\alpha \circ \mathcal {F}\) still yields a convergent fixed-point iteration, but with a modified fixed point, whose distance to the original fixed point of \(\mathcal {F}\) is quantified by the following result.

Lemma 4.1

Assuming (4.1), let \(\mathbf {u}^*\) be the unique fixed point of \(\mathcal {F}\). Then, for any \(\alpha >0\), there exists a uniquely determined \(\mathbf {u}^\alpha \) such that \(\mathbf {u}^\alpha = {\mathbf {S}}_\alpha \bigl ( \mathcal {F}(\mathbf{u}^\alpha ) \bigr )\), which satisfies
$$\begin{aligned} ( 1+ \rho )^{-1} || {\mathbf {S}}_\alpha (\mathbf{u}^*)-\mathbf{u}^*|| \le || \mathbf {u}^\alpha - \mathbf {u}^* || \le (1-\rho )^{-1} || {\mathbf {S}}_\alpha (\mathbf{u}^*)-\mathbf{u}^*||. \end{aligned}$$
(4.2)
Moreover, for any given \(\mathbf {u}_0\), for \(\mathbf {u}_{k+1} {:}{=} {\mathbf {S}}_\alpha \bigl ( \mathcal {F}(\mathbf{u}_k) \bigr )\), one has
$$\begin{aligned} ||\mathbf {u}_k - \mathbf {u}^\alpha || \le \rho ^k ||\mathbf {u}_0 - \mathbf {u}^\alpha ||. \end{aligned}$$

Proof

By the non-expansiveness of \({\mathbf {S}}_\alpha \), the operator \(\mathcal {G}{:}{=} {\mathbf {S}}_\alpha \circ \mathcal {F}\) is a contraction. The existence and uniqueness of \(\mathbf{u}^\alpha \), as well as the stated properties of the iteration, thus follow from the Banach fixed-point theorem. Let \(\mathbf {e}^\alpha {:}{=} {\mathcal {G}}(\mathbf{u}^\alpha ) - {\mathcal {G}}(\mathbf{u}^*)\). Then, since \(\mathcal {G}(\mathbf{u}^\alpha ) = \mathbf{u}^\alpha \), one has
$$\begin{aligned} \mathbf{u}^\alpha - \mathbf{u}^* = {\mathcal {G}}(\mathbf{u}^*) - \mathbf{u}^* + \mathbf {e}^\alpha . \end{aligned}$$
Combining this with the observation
$$\begin{aligned} ||\mathbf {e}^\alpha || = ||{\mathcal {G}}(\mathbf{u}^\alpha ) - {\mathcal {G}}(\mathbf{u}^*)|| \le ||\mathcal {F}(\mathbf{u}^\alpha ) - \mathcal {F}(\mathbf{u}^*)|| \le \rho ||\mathbf{u}^\alpha -\mathbf{u}^*||, \end{aligned}$$
where we have again used that \({\mathbf {S}}_\alpha \) is non-expansive, yields
$$\begin{aligned} ||\mathbf{u}^\alpha - \mathbf{u}^*|| \le ||{\mathcal {G}}(\mathbf{u}^*) - \mathbf{u}^*|| + \rho ||\mathbf{u}^\alpha - \mathbf{u}^*|| \end{aligned}$$
as well as
$$\begin{aligned} ||{\mathcal {G}}(\mathbf{u}^*) - \mathbf{u}^*|| \le ||\mathbf{u}^\alpha - \mathbf{u}^*|| + \rho ||\mathbf{u}^\alpha - \mathbf{u}^*||. \end{aligned}$$
Finally, noting that \({\mathcal {G}}(\mathbf{u}^*) = {\mathbf {S}}_\alpha (\mathbf{u}^*)\) since \(\mathcal {F}(\mathbf{u}^*)=\mathbf{u}^*\), we arrive at (4.2). \(\square \)

For fixed thresholding parameter \(\alpha \), the thresholded Richardson iteration thus converges, at the same rate \(\rho \) as the unperturbed Richardson iteration, to a modified solution \(\mathbf{u}^\alpha \). Its distance to the true solution \(\mathbf{u}^*\) is proportional, uniformly in \(\alpha \), to \(||{\mathbf {S}}_\alpha (\mathbf{u}^*)-\mathbf{u}^*||\), that is, to the error of thresholding \(\mathbf{u}^*\). Note that Lemma 4.1 in fact also holds with \({\mathbf {S}}_\alpha \) replaced by an arbitrary non-expansive mapping on \(\mathcal {H}\).

4.1 A Priori Choice of Thresholding Parameters

In order to obtain convergence to \(\mathbf{u}^*\), instead of working with a fixed \(\alpha \), we will instead consider the iteration
$$\begin{aligned} \mathbf {u}_{k+1} = {\mathbf {S}}_{\alpha _k} \bigl ( \mathcal {F}(\mathbf{u}_k) \bigr ), \end{aligned}$$
(4.3)
where we choose \(\alpha _k\) with \(\alpha _k \rightarrow 0\). The central question is now how one can obtain a suitable choice of this sequence; clearly, if the \(\alpha _k\) decrease too slowly, this will hamper the convergence of the iteration, whereas \(\alpha _k\) that decrease too quickly may lead to very large tensor ranks of the iterates.
In principle, if the decay of the sequences \(\sigma _t(\mathbf{u}^*)\) is known, for instance \(\sigma _t(\mathbf{u}^*) \in \ell ^{p,\infty }\), then Proposition 3.6 immediately gives us a choice of \(\alpha _k\) that ensures convergence to \(\mathbf{u}^*\) with almost the unperturbed rate \(\rho \). To this end, observe that
$$\begin{aligned} || \mathbf{u}_{k+1} - \mathbf{u}^* ||\le & {} ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k} || + ||\mathbf{u}^{\alpha _k} - \mathbf{u}^*|| \nonumber \\\le & {} \rho ||\mathbf{u}_k - \mathbf{u}^* || + ( 1 + \rho ) ||\mathbf{u}^{\alpha _k} - \mathbf{u}^*|| \nonumber \\\le & {} \rho ||\mathbf{u}_{k} - \mathbf{u}^* || + \frac{1+\rho }{1-\rho } \sum _{t} d^{\alpha _k}_t (\mathbf{u}^*), \end{aligned}$$
(4.4)
and based on Proposition 3.6, we can adjust \(\alpha _k\) in every step so as to balance the decrease in the two terms on the right-hand side of (4.4). We next give such choices of \(\alpha _k\) for the two cases in Proposition 3.6. To simplify the statements, we always take \(\mathbf{u}_0 {:}{=} 0\) and assume that \(\alpha _0\) is sufficiently large to ensure
$$\begin{aligned} \mathbf{u}_1 = {\mathbf {S}}_{\alpha _0} ( \omega {\mathbf {f}}) = 0, \end{aligned}$$
(4.5)
which implies that \(\mathbf {u}^{\alpha _0} = 0\).

Proposition 4.2

Let \(\sigma _t (\mathbf{u}^*) \in \ell ^{p,\infty }, t=1,\ldots ,E\), with \(0<p<2\). Let \(\mathbf{u}_0 {:}{=} 0\).
  1. (i)
    Then, for the choice \(\alpha _k {:}{=} (\rho ^{k+1} c_0)^\frac{2}{2-p}\) in the iteration (4.3), with \(c_0>0\) such that (4.5) holds, we have
    $$\begin{aligned} ||\mathbf{u}_{k} - \mathbf{u}^*|| \le C_1 E \max _t |\sigma _t(\mathbf{u}^*)|^{p/2}_{\ell ^{p,\infty }} \, k \, \rho ^k, \end{aligned}$$
    where \(C_1\) depends on \(\rho \) and p. Furthermore, for any \(\tilde{\rho }\) such that \(\rho < {\tilde{\rho }} < 1\), with \(\alpha _k {:}{=} ({\tilde{\rho }}^{k+1} c_0)^\frac{2}{2-p}\), we have
    $$\begin{aligned} ||\mathbf{u}_{k} - \mathbf{u}^*|| \le C_2 E \max _t |\sigma _t(\mathbf{u}^*)|^{p/2}_{\ell ^{p,\infty }} (\tilde{\rho }-\rho )^{-1} {\tilde{\rho }}^{k+1} , \end{aligned}$$
    (4.6)
    where \(C_2\) depends on \(\rho \) and p.
     
  2. (ii)
    If \(\sigma _{t,k}(\mathbf{u}^*) \le C e^{- c k^\beta }\) with \(C, c , \beta > 0\), for the choice \(\alpha _k {:}{=} \rho ^{k+1} c_0\), we have
    $$\begin{aligned} ||\mathbf{u}_{k} - \mathbf{u}^*|| \lesssim E k^{1 + {\frac{1}{2\beta }}} \, \rho ^k, \end{aligned}$$
    and with \(\alpha _k {:}{=} {\tilde{\rho }}^{k+1} c_0\), we have instead
    $$\begin{aligned} ||\mathbf{u}_{k} - \mathbf{u}^*|| \lesssim E \,k^{ {\frac{1}{2\beta }}} \, {\tilde{\rho }}^k, \end{aligned}$$
    (4.7)
    where the constant depends on \(({\tilde{\rho }} - \rho )^{-1}\).
     

Proof

Using (4.4) and Proposition 3.6(i), we obtain
$$\begin{aligned} ||\mathbf{u}_k - \mathbf{u}^*|| \le \rho ^k || \mathbf{u}^* || + C_1 E \max _t |\sigma _t(\mathbf{u}^*)|^{\frac{p}{2}}_{\ell ^{p,\infty }} \sum _{i=0}^{k-1} \rho ^{k-i-1} \alpha _i^{1-\frac{p}{2}}. \end{aligned}$$
Note that with our choice of \(\alpha _k\), we have \(\alpha _i^{1-\frac{p}{2}} = \rho ^{i+1} c_0\). Using in addition that \(||\mathbf{u}^*|| = || \mathbf{u}^{\alpha _0} - \mathbf{u}^* || \lesssim ||{\mathbf {S}}_{\alpha _0}(\mathbf{u}^*)-\mathbf{u}^*||\) due to (4.5), we obtain the first statement. For the second choice of \(\alpha _k\), we obtain
$$\begin{aligned} ||\mathbf{u}_k - \mathbf{u}^*|| \le \Bigl ( \theta ^k ||\mathbf{u}^*|| + C_1 E \max _t |\sigma _t(\mathbf{u}^*)|^{\frac{p}{2}}_{\ell ^{p,\infty }} c_0 \sum _{i=0}^{k-1} \theta ^{k-1-i} \Bigr ) {\tilde{\rho }}^k, \quad \theta {:}{=} \frac{\rho }{{\tilde{\rho }}} < 1, \end{aligned}$$
and the corresponding statement thus follows.
Under the second set of assumptions, we proceed analogously based on Proposition 3.6(ii), which for \(\alpha _k {:}{=} \rho ^{k+1} c_0\) yields
$$\begin{aligned} ||\mathbf{u}_k - \mathbf{u}^*|| \lesssim \rho ^k + E \sum _{i=0}^{k-1} \rho ^{k-i-1} (1 + |\ln c_0| + i |\ln \rho |)^{\frac{1}{2\beta }} \rho ^{i+1}, \end{aligned}$$
where the modified power of k in the assertion thus arises due to the logarithmic term in (3.5). For \(\alpha _k {:}{=} {\tilde{\rho }}^{k+1} c_0\), with \(\theta \) as above,
$$\begin{aligned} ||\mathbf{u}_k - \mathbf{u}^*|| \lesssim \Bigl ( \theta ^k + E \sum _{i=0}^{k-1} \theta ^{k-i-1} (1 + i |\ln {\tilde{\rho }}|)^{\frac{1}{2\beta }} \Bigr ) {\tilde{\rho }}^{i+1} \lesssim E k^{\frac{1}{2\beta }} {\tilde{\rho }}^k. \end{aligned}$$
\(\square \)

In summary, in this idealized setting with full knowledge of the decay of \(||{\mathbf {S}}_\alpha (\mathbf{u}^*) - \mathbf{u}^*||\) with respect to \(\alpha \), the above choices ensure convergence of the iteration with any asymptotic rate \(\tilde{\rho }> \rho \).

4.2 Rank Estimates

We now give estimates for the ranks of the iterates that can arise in the course of the iteration, assuming that the \(\alpha _k\) are chosen as in Proposition 4.2.

For the proof, we will use the following lemma, which is a direct adaptation of [11, Lemma5.1], where the same argument was applied to hard thresholding of sequences; it was restated for soft thresholding of sequences (with the same proof) in [14].

Lemma 4.3

Let \(\mathbf{v},\mathbf{w}\in \mathcal {H}, \alpha > 0\), and \(\varepsilon >0\) such that \(||\mathbf{v}-\mathbf{w}||\le \varepsilon \).
  1. (i)
    If \(\sigma _t(\mathbf{v})\in \ell ^{p,\infty }\) with \(0<p<2\) for all \(t\in \{1,\ldots ,E\}\), then
    $$\begin{aligned} {{\mathrm{rank}}}_t \bigl ({\mathbf {S}}_\alpha (\mathbf{w})\bigr ) \le \frac{4 \varepsilon ^2}{\alpha ^2} + C_p |\sigma _t(\mathbf{v})|^p_{\ell ^{p,\infty }} \alpha ^{-p}. \end{aligned}$$
     
  2. (ii)
    If \(\sigma _{t,j}(\mathbf{v}) \le C e^{-c j^\beta }\) for \(j\in {\mathbb {N}}\) with \(C, c, \beta > 0\), then
    $$\begin{aligned} {{\mathrm{rank}}}_t \bigl ({\mathbf {S}}_\alpha (\mathbf{w})\bigr ) \le \frac{4 \varepsilon ^2}{\alpha ^2} + \bigl ( c^{-1} \ln (2C\alpha ^{-1}) \bigr )^{1/\beta }. \end{aligned}$$
     

Proof

Note first that as an immediate consequence of Theorem 2.1, for each t we have
$$\begin{aligned} ||\sigma _t(\mathbf{v}) - \sigma _t(\mathbf{w})||= & {} \Bigl ( \sum _i |\sigma _{t,i}(\mathbf{v})-\sigma _{t,i}(\mathbf{w})|^2\Bigr )^{\frac{1}{2}}\\&\quad \le ||\mathcal {M}_t(\mathbf{v}) - \mathcal {M}_t(\mathbf{w})||_{\mathrm {HS}}= ||\mathbf{v}- \mathbf{w}|| \le \varepsilon . \end{aligned}$$
Furthermore, Lemma 3.3 yields \({{\mathrm{rank}}}_t({\mathbf {S}}_\alpha (\mathbf{w})) \le {{\mathrm{rank}}}_t(S_{t,\alpha }(\mathbf{w}))\), and it thus suffices to estimate the latter.
The first inequality in the statement now follows with the same argument as in [11], which we include for the convenience of the reader. We abbreviate \(a{:}{=} \sigma _t(\mathbf{v}), b{:}{=} \sigma _t(\mathbf{w})\). Let \(\mathcal {I}_1 {:}{=} \{ i:b_i \ge \alpha , a_i > \alpha /2\}\) and \(\mathcal {I}_2 {:}{=} \{ i :b_i \ge \alpha , a_i \le \alpha /2\}\). Then,
$$\begin{aligned} \Bigl ( \frac{\alpha }{2} \Bigr )^2 \#\mathcal {I}_2 \le \sum _{i\in \mathcal {I}_2} |a_i - b_i|^2 \le \varepsilon ^2 \end{aligned}$$
as well as
$$\begin{aligned} \#\mathcal {I}_1 \le \#\{ i:a_i > \alpha /2\} \le C_p |\sigma _t(\mathbf{v})|^p_{\ell ^{p,\infty }} \alpha ^{-p}, \end{aligned}$$
which proves the first statement, since \( {{\mathrm{rank}}}_t(S_{t,\alpha }(\mathbf{w})) \le \#\mathcal {I}_1 + \#\mathcal {I}_2 \). To obtain the second inequality, we use Proposition 3.6(ii) to estimate \(\#\mathcal {I}_1\) in an analogous way. \(\square \)

Theorem 4.4

Let \(\rho < {\tilde{\rho }} < 1, \mathbf{u}_0 {:}{=} 0\), and \(\varepsilon _k {:}{=} {\tilde{\rho }}^k\).
  1. (i)
    If \(\sigma _t (\mathbf{u}^*) \in \ell ^{p,\infty }, t=1,\ldots ,E\), with \(0<p<2\), then for the choice \(\alpha _k {:}{=} ({\tilde{\rho }}^{k+1} c_0)^\frac{2}{2-p}\) in the iteration (4.3), with \(c_0>0\) such that (4.5) holds, we have
    $$\begin{aligned} ||\mathbf{u}_k -\mathbf{u}^*|| \lesssim d \varepsilon _k, \qquad \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{u}_k) \lesssim d^2 \varepsilon _k^{-\frac{1}{s}}, \end{aligned}$$
    (4.8)
    where \(s = \textstyle \frac{1}{p} - \frac{1}{2}\).
     
  2. (ii)
    If \(\sigma _{t,j}(\mathbf{u}^*) \le C e^{- c j^\beta }\), for \(j\in {\mathbb {N}}\) and \(t=1,\ldots ,E\), with \(C, c, \beta > 0\), for the choice \(\alpha _k {:}{=} {\tilde{\rho }}^{k+1} c_0\), we have
    $$\begin{aligned} ||\mathbf{u}_{k} - \mathbf{u}^*||\lesssim & {} d \bigl ( 1+ |\log \varepsilon _k| \bigr )^{\frac{1}{2\beta }} \varepsilon _k, \qquad \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{u}_k)\nonumber \\\lesssim & {} d^2 \bigl ( 1+ |\log \varepsilon _k|\bigr )^{\frac{1}{\beta }}. \end{aligned}$$
    (4.9)
     

Proof

Recall that \(E = 2d -3\). The estimates \(||\mathbf{u}_k - \mathbf{u}^*||\lesssim E \varepsilon _k \) are shown in (4.6) and (4.7) of Proposition 4.2. To obtain the corresponding estimates for the ranks, we use Lemma 4.3. Note that with \(\mathbf{w}_k {:}{=} \mathcal {F}(\mathbf{u}_{k-1})\), we have \(\mathbf{u}_k = {\mathbf {S}}_{\alpha _{k-1}}(\mathbf{w}_k)\) and, by contractivity of \(\mathcal {F}\),
$$\begin{aligned} ||\mathbf{w}_k - \mathbf{u}^*|| \le \rho ||\mathbf{u}_{k-1} - \mathbf{u}^*|| \lesssim E \varepsilon _{k-1}. \end{aligned}$$
In the first case, for each t, Lemma 4.3 gives
$$\begin{aligned} {{\mathrm{rank}}}_t({\mathbf {S}}_{\alpha _{k-1}}(\mathbf{w}_k)) \lesssim \frac{(E\varepsilon _k)^2}{\alpha _{k-1}^2} + \alpha _{k-1}^{-p} \lesssim E^2 \bigl ( {\tilde{\rho }}^{1 -\frac{2}{2-p}}\bigr )^{2k} + {\tilde{\rho }}^{-\frac{2pk}{2-p}}. \end{aligned}$$
Noting that \(\bigl ({\tilde{\rho }}^{1 -\frac{2}{2-p}}\bigr )^{2k} = {\tilde{\rho }}^{-\frac{2pk}{2-p}} = \varepsilon _k^{-\frac{1}{s}}\), we obtain the first assertion. In the second case, we have \(E\varepsilon _k / \alpha _k \lesssim E \bigl ( 1 + |\log \varepsilon _k| \bigr )^{\frac{1}{2\beta } }\), and the lemma thus yields
$$\begin{aligned} {{\mathrm{rank}}}_t({\mathbf {S}}_\alpha (\mathbf{w}_k)) \lesssim E^2 \bigl (1 + |\log \varepsilon _k| \bigr )^{\frac{1}{\beta }} + \bigl ( 1 + |\log {\tilde{\rho }}^k c_0| \bigr )^{\frac{1}{\beta }} \lesssim E^2 \bigl (1 + |\log \varepsilon _k| \bigr )^{\frac{1}{\beta }}. \end{aligned}$$
\(\square \)

Remark 4.5

Note that the asymptotic statement in (4.9) can be simplified by considering any \({\hat{\rho }}\) with \({\tilde{\rho }}<{\hat{\rho }}<1\) and \({\hat{\varepsilon }}_k {:}{=} {\hat{\rho }}^k\), for which we obtain
$$\begin{aligned} ||\mathbf{u}_{k} - \mathbf{u}^*|| \lesssim d {\hat{\varepsilon }}_k, \qquad \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{u}_k) \lesssim d^2 \bigl ( 1+ |\log {\hat{\varepsilon }}_k|\bigr )^{\frac{1}{\beta }} \end{aligned}$$
where the constants now depend also on \({\hat{\rho }}\).

Theorem 4.4 yields quasi-optimal rank bounds in the sense discussed in Sect. 1.1. We comment further on this point after stating our main result Theorem 5.1, which is of the same type but concerns a practical algorithm, in the next section.

5 A Posteriori Choice of Parameters

The results of the previous section lead to the question whether estimates as in Theorem 4.4 can still be recovered when a priori knowledge of the decay of the sequences \(\sigma _t(\mathbf{u}^*)\) is not available. We thus consider now a modified scheme that adjusts the \(\alpha _k\) automatically without using such information on \(\mathbf{u}^*\), but still yields quasi-optimal rank estimates as in Theorem 4.4 for both cases considered there.

The design of such a method is more problem-specific than the general considerations in the previous section, and here, we thus restrict ourselves to linear operator equations \( {\mathcal {A}} \mathbf{u}= {\mathbf {f}} \) with \({\mathcal {A}}\) symmetric, bounded, and elliptic. We assume to have \(\gamma , \varGamma >0\) such that
$$\begin{aligned} \gamma ||\mathbf{v}||^2 \le \langle {\mathcal {A}}\mathbf{v},\mathbf{v}\rangle \le \varGamma ||\mathbf{v}||^2, \quad \mathbf{v}\in \mathcal {H}, \end{aligned}$$
(5.1)
that is, the spectrum of \({\mathcal {A}}\) is contained in \([\gamma ,\varGamma ]\) and \(\gamma ^{-1}\varGamma \) is an upper bound for the condition number of \({\mathcal {A}}\). The choice \(\omega {:}{=} 2 / (\gamma + \varGamma )\) then yields
$$\begin{aligned} ||\mathrm {Id}- \omega {\mathcal {A}}|| \le \frac{\varGamma - \gamma }{\varGamma + \gamma } =: \rho < 1, \end{aligned}$$
(5.2)
and the results of the previous section apply with \(\mathcal {F}(\mathbf{v}) {:}{=} \mathbf{v}- \omega ({\mathcal {A}} \mathbf{v}- {\mathbf {f}})\).
In order to be able to obtain estimates for the ranks of iterates as in Theorem 4.4, the method in Algorithm 2 is constructed such that whenever \(\alpha _k\) is decreased in the iteration,
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}^*|| \le C ||\mathbf{u}^{\alpha _k} - \mathbf{u}^*|| \end{aligned}$$
(5.3)
holds with some fixed constant \(C>1\). It will be established in what follows that the validity of such an estimate ensures that \(\alpha _k\) never becomes too small in relation to the corresponding current error \(||\mathbf{u}_k - \mathbf{u}^*||\). A bound of the form (5.3) is ensured by the condition in line 5 of Algorithm 2, which is explained in more detail in the proof of Theorem 5.1 below.

Note that Algorithm 2 only requires—besides a hierarchical tensor representation of \({\mathbf {f}}\) and the action of \({\mathcal {A}}\) on such representations—bounds \(\gamma , \varGamma \) on the spectrum of \({\mathcal {A}}\), certain quantities derived from these, as well as constants \(\nu \) and \(\theta \) that can be adjusted arbitrarily. The following is the main result of this work.

Theorem 5.1

The procedure \({{\mathrm{\textsc {STSolve}}}}\) given in Algorithm 2 produces \(\mathbf{u}_\varepsilon \) satisfying \(||\mathbf{u}_\varepsilon - \mathbf{u}^*|| \le \varepsilon \) in finitely many steps.
  1. (i)
    If \(\sigma _t (\mathbf{u}^*) \in \ell ^{p,\infty }, t=1,\ldots ,E\), with \(0<p<2\), then there exists \({\tilde{\rho }} \in (0,1)\) such that with \(\varepsilon _k {:}{=} {\tilde{\rho }}^k\) and \(s = \frac{1}{p} - \frac{1}{2}\), one has
    $$\begin{aligned} ||\mathbf{u}_k -\mathbf{u}^*|| \lesssim d \varepsilon _k, \qquad \max _{t=1,\ldots ,E}{{\mathrm{rank}}}_t(\mathbf{u}_k) \lesssim d^2 \max _{t=1,\ldots ,E} |\sigma _t(\mathbf{u}^*)|_{\ell ^{p,\infty }}^{\frac{1}{s}} \varepsilon _k^{-\frac{1}{s}}. \end{aligned}$$
    (5.4)
    The constants depend on \(\gamma , \varGamma , \theta , \nu , \alpha _0\), and p.
     
  2. (ii)
    If \(\sigma _{t,j}(\mathbf{u}^*) \le C e^{- c j^\beta }\), for \(j\in {\mathbb {N}}\) and \(t=1,\ldots ,E\), with \(C, c, \beta > 0\), then there exists \({\tilde{\rho }} \in (0,1)\) such that with \(\varepsilon _k {:}{=} {\tilde{\rho }}^k\), one has
    $$\begin{aligned} ||\mathbf{u}_{k} - \mathbf{u}^*|| \lesssim d \varepsilon _k, \qquad \max _{t=1,\ldots ,E}{{\mathrm{rank}}}_t(\mathbf{u}_k) \lesssim d^2 \bigl ( 1+ |\ln \varepsilon _{k}| \bigr )^{\frac{1}{\beta }}. \end{aligned}$$
    (5.5)
    The constants depend on \(\gamma , \varGamma , \theta , \nu , \alpha _0\), and on Cc, and \(\beta \).
     

Remark 5.2

The estimate (5.4) implies, as discussed in Sect. 1.1,
$$\begin{aligned} ||\mathbf{u}_k -\mathbf{u}^*|| \le B d^{1+2s} \max _{t=1,\ldots ,E} |\sigma _t(\mathbf{u}^*)|_{\ell ^{p,\infty }} \Bigl ( \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{u}_k)\Bigr )^{-s}, \end{aligned}$$
where \(B>0\) depends only on the spectral bounds \(\gamma ,\varGamma \), on p, and on the parameters of the iteration. Thus, we also have quasi-optimality of approximation errors. Note that the arising power of d does not depend on further properties of \({\mathcal {A}}\). In the case of (5.5), we obtain
$$\begin{aligned} ||\mathbf{u}_k -\mathbf{u}^*|| \le \bar{C} d \exp \biggl ( - \bar{c} d^{-2\beta } \Bigl (\max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{u}_k)\Bigr )^{\beta } \biggr ) \end{aligned}$$
with \(\bar{C}, \bar{c}>0\). Hence, as noted above, in this case, rank bounds that are optimal up to a multiplicative constant preserve the exponent \(\beta \) in the corresponding error bounds, but both factors Cc are modified.

Remark 5.3

The choice of \(\nu \) and \(\theta \) enters only into the value of \({\tilde{\rho }}\) and into the multiplicative constants in (5.4) and (5.5), but does not influence the asymptotic dependence on d and \(\varepsilon _k\). The results of Proposition 4.2, however, suggest to choose \(\theta \) smaller than \(\rho \) in order to avoid unnecessarily slowing down the convergence of the iteration. Thus, one may for instance simply set \(\nu =\frac{1}{2}, \theta =\frac{\rho }{2}\), although a further adjustment may improve the quantitative behavior of the iteration for a given problem.

In the proof of Theorem 5.1, we will use the following technical lemma, which limits the decay of the soft thresholding error as the thresholding parameter is decreased.

Lemma 5.4

Let \(\mathbf{v}\ne 0\), then \( d^{\alpha }_t(\mathbf{v}) \le \theta ^{-1} d^{\theta \alpha }_t (\mathbf{v}) , t=1,\ldots ,E\), for all \(\alpha >0, \theta \in (0,1)\).

Proof

For the proof, we omit the dependence of quantities on \(\mathbf{v}\). We clearly have \(r_{t, \theta \alpha } \ge r_{t,\alpha }\) and \(\tau _{t, \theta \alpha } \le \tau _{t,\alpha }\). Furthermore, \( \tau _{t,\alpha }^2 - \tau _{t,\theta \alpha }^2 \le \alpha ^2 ( r_{t, \theta \alpha } - r_{t, \alpha }) \), and consequently,
$$\begin{aligned} \biggl ( \frac{d^{\alpha }_t}{d^{\theta \alpha }_t }\biggr )^2 = \frac{\alpha ^2 r_{t,\alpha } + \tau _{t,\alpha }^2 }{(\theta \alpha )^2 r_{t,\theta \alpha } + \tau _{t,\theta \alpha }^2 } \le \frac{\alpha ^2 r_{t,\alpha } + \tau ^2_{t,\theta \alpha } + \alpha ^2 ( r_{t,\theta \alpha } - r_{t,\alpha }) }{(\theta \alpha )^2 r_{t,\theta \alpha } + \tau ^2_{t,\theta \alpha }} \le \theta ^{-2}. \end{aligned}$$
\(\square \)

Proof of Theorem 5.1

Step 1 We first show that the condition
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}_k|| \le \frac{(1 - \rho ) \nu }{ \varGamma \rho } ||\mathbf {r}_{k+1}|| \end{aligned}$$
(5.6)
in line 5 of the algorithm is always satisfied after a finite number of steps.
The combination of the first inequality of (3.2) in Lemma 3.4 and the first inequality in (4.2) of Lemma 4.1 shows
$$\begin{aligned} ( 1+ \rho )^{-1} \max _t d^\alpha _t(\mathbf{u}^*) \le ||\mathbf{u}^\alpha - \mathbf{u}^*||. \end{aligned}$$
Thus, we always have \(||\mathbf{u}^\alpha - \mathbf{u}^*|| > 0\) if \(\alpha >0\), unless \(\mathbf{u}^* = 0\) and hence \({\mathbf {f}} = 0\). In the latter case, however, the algorithm stops immediately, and we can thus assume that \(\mathbf{u}^\alpha \ne \mathbf{u}^*\) for any positive \(\alpha \).
If \(\alpha _k = \ldots = \alpha _0\), we have on the one hand
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}_k|| \le ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _0} || + ||\mathbf{u}_k - \mathbf{u}^{\alpha _0}|| \le \rho ^k ( 1 + \rho ) ||\mathbf{u}_0 - \mathbf{u}^{\alpha _0}||, \end{aligned}$$
(5.7)
and on the other hand, we similarly obtain
$$\begin{aligned} \gamma ^{-1} ||\mathbf {r}_{k+1}|| \ge ||\mathbf{u}_{k+1} - \mathbf{u}^*||\ge & {} ||\mathbf{u}^{\alpha _0} - \mathbf{u}^* || - \rho ^{k+1} ||\mathbf{u}_0 - \mathbf{u}^{\alpha _0} || \nonumber \nonumber \\\ge & {} (1 - \rho ^{k+1}) ||\mathbf{u}^{\alpha _0} - \mathbf{u}^*|| - \rho ^{k+1} ||\mathbf{u}_0 - \mathbf{u}^*||. \end{aligned}$$
(5.8)
Thus, the right-hand side in (5.7) converges to zero, whereas the right-hand side in (5.8) is bounded away from zero for sufficiently large k. Hence, (5.6) holds with \(k = J_0 -1\) for some \(J_0 \in {\mathbb {N}}\), and we assume this to be the minimum integer with this property. The thresholding parameter is then decreased for the following iteration, that is, \(\alpha _{J_0} = \theta \alpha _{J_0 -1} = \theta \alpha _{0}\). As in (4.4), for \(k < J_0\), we obtain
$$\begin{aligned} || \mathbf{u}_{k+1} - \mathbf{u}^* || \le \rho ^{k+1} ||\mathbf{u}_0 - \mathbf{u}^*|| + (1 + \rho ^{k+1}) ||\mathbf{u}^{\alpha _0} - \mathbf{u}^*||. \end{aligned}$$
The same arguments then apply with \(\alpha _{0}\) replaced by \(\alpha _{J_0}\) and \(\mathbf{u}_0\) by \(\mathbf{u}_{J_0}\). Thus, \(\alpha _k\) will always be decreased after a finite number of steps.
Step 2 To show convergence of the \(\mathbf{u}_k\), we first observe that by our requirement that \(\alpha _0 \ge E^{-1} \omega ||{\mathbf {f}}||\) (which is in fact not essential for the execution of the iteration), we actually have \(\mathbf{u}_1=0\) and hence \(\mathbf{u}^{\alpha _0} = 0\), implying also \(J_0=1\). In particular,
$$\begin{aligned} || \mathbf{u}_{n} - \mathbf{u}^* || \le || \mathbf{u}^* - \mathbf{u}^{\alpha _0} ||, \quad 0 \le n \le J_0 = 1. \end{aligned}$$
(5.9)
We next investigate the implications of the condition (5.6) on the further iterates. Note first that
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}^*|| - ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _{k}}|| \le ||\mathbf{u}^* - \mathbf{u}^{\alpha _k}||. \end{aligned}$$
(5.10)
The standard error estimate for contractive mappings, combined with (5.6) and (5.1), gives
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k}|| \le \frac{\rho }{1-\rho } ||\mathbf{u}_{k+1} - \mathbf{u}_k|| \le \frac{\nu }{\varGamma }||\mathbf {r}_{k+1}|| \le \nu ||\mathbf{u}_{k+1} - \mathbf{u}^*||. \end{aligned}$$
(5.11)
Inserting the latter into (5.10), we thus obtain
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}^*||\le & {} (1 - \nu )^{-1} \bigl ( ||\mathbf{u}_{k+1} - \mathbf{u}^*|| - ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k}|| \bigr ) \nonumber \nonumber \\\le & {} (1 - \nu )^{-1} ||\mathbf{u}^* - \mathbf{u}^{\alpha _k}||. \end{aligned}$$
(5.12)
We introduce the following notation that groups iterates according to the corresponding values of \(\alpha _k\): For each \(i \in {\mathbb {N}}_0\), let \(\eta _{i} {:}{=} \theta ^i \alpha _0\). With \(\mathbf{u}_{0,0} {:}{=} \mathbf{u}_0 = 0\) and \(\mathbf {r}_{0,0} {:}{=} - {\mathbf {f}}\), iterates are produced according to
$$\begin{aligned} \mathbf{w}_{i,j+1} {:}{=} \mathbf{u}_{i,j} - \omega \mathbf {r}_{i,j} , \quad \mathbf{u}_{i,j + 1} {:}{=} \mathbf {S}_{\eta _{i}} (\mathbf{w}_{i,j + 1}), \quad \mathbf {r}_{i,j+1} {:}{=} {\mathcal {A}} \mathbf{u}_{i,j+1} - {\mathbf {f}}, \end{aligned}$$
(5.13)
where the index i is increased each time that condition (5.6) is satisfied. For each i, consistently with the previous definition of \(J_0\), we define \(J_i\) as the last index of an iterate produced with the value \(\eta _{i}\), which means that \(\mathbf{u}_{i+1,0} = \mathbf{u}_{i,J_i}\) and, as a consequence of (5.12),
$$\begin{aligned} || \mathbf{u}_{i+1, 0} - \mathbf{u}^* || = || \mathbf{u}_{i, J_i} - \mathbf{u}^* || \le (1-\nu )^{-1} ||\mathbf{u}^* - \mathbf{u}^{\eta _{i}} ||, \quad i \ge 0. \end{aligned}$$
(5.14)
For \(i \ge 0\) and \(j=0,\ldots ,J_i\), with (5.14), we obtain
$$\begin{aligned} ||\mathbf{u}_{i,j} - \mathbf{u}^*||\le & {} ||\mathbf{u}_{i,j} - \mathbf{u}^{\eta _i} || + || \mathbf{u}^{\eta _i} - \mathbf{u}^*|| \nonumber \nonumber \\\le & {} \rho ^j ||\mathbf{u}_{i,0} - \mathbf{u}^*|| + (1 + \rho ^j) || \mathbf{u}^{\eta _i} - \mathbf{u}^*|| \nonumber \nonumber \\\le & {} (1 - \nu )^{-1} \rho ^j ||\mathbf{u}^{\eta _{i-1}} - \mathbf{u}^*|| + (1 + \rho ^j) || \mathbf{u}^{\eta _i} - \mathbf{u}^*||, \end{aligned}$$
(5.15)
where we have used (5.9) in the case \(i=0\). By Proposition 3.6, this implies in particular that, in our original notation, \(\mathbf{u}_k \rightarrow \mathbf{u}^*\).
Step 3 Our next aim is to estimate the values of \(J_i\). We have already established that \(J_0 = 1\). In order to estimate \(J_i\) for \(i>0\), we use (5.7) and (5.8) to obtain
$$\begin{aligned} \frac{ ||\mathbf{u}_{i,j+1} - \mathbf{u}_{i,j}|| }{ ||\mathbf {r}_{i,j+1} || } \le \frac{ \rho ^j \gamma ^{-1} (1+\rho ) || \mathbf{u}_{i,0} - \mathbf{u}^{\eta _i}|| }{ ||\mathbf{u}^{\eta _i} - \mathbf{u}^*|| - \rho ^{j+1} ||\mathbf{u}_{i,0} - \mathbf{u}^{\eta _i}|| } \end{aligned}$$
for j sufficiently large. Thus, (5.6) follows if the two conditions
$$\begin{aligned} \rho ^j \gamma ^{-1} (1+\rho ) \frac{||\mathbf{u}_{i,0} - \mathbf{u}^{\eta _i}||}{||\mathbf{u}^{\eta _i} - \mathbf{u}^*||} \le \frac{(1-\rho ) \nu }{2 \varGamma \rho },\quad \rho ^{j+1} \frac{||\mathbf{u}_{i,0} - \mathbf{u}^{\eta _i}||}{||\mathbf{u}^{\eta _i} - \mathbf{u}^*||} \le \frac{1}{2} \end{aligned}$$
hold. These are guaranteed if
$$\begin{aligned} j \ge |\ln \rho |^{-1} \biggl ( C(\gamma ,\varGamma ,\nu ) + \ln \frac{||\mathbf{u}_{i,0} - \mathbf{u}^{\eta _i}||}{||\mathbf{u}^{\eta _i} - \mathbf{u}^*||} \biggr ) \end{aligned}$$
(5.16)
with some constant \(C(\gamma ,\varGamma ,\nu )\ge 0\). By (5.14),
$$\begin{aligned} \ln \frac{||\mathbf{u}_{i,0} - \mathbf{u}^{\eta _i}||}{||\mathbf{u}^{\eta _i} - \mathbf{u}^*||} \le \ln \biggl ( 1 + (1- \nu )^{-1} \frac{||\mathbf{u}^{\eta _{i-1}} - \mathbf{u}^*||}{||\mathbf{u}^{\eta _i} - \mathbf{u}^*||} \biggr ), \end{aligned}$$
and by Lemma 3.4 and Lemma 4.1,
$$\begin{aligned} \frac{||\mathbf{u}^{\eta _{i-1}} - \mathbf{u}^*||}{||\mathbf{u}^{\eta _i} - \mathbf{u}^*||}\le & {} \frac{(1+\rho ) ||\mathbf {S}_{\eta _{i-1}}(\mathbf{u}^*) - \mathbf{u}^*||}{(1-\rho ) ||\mathbf {S}_{\eta _{i}}(\mathbf{u}^*) - \mathbf{u}^*|| }\\\le & {} \frac{(1+\rho )}{(1-\rho )} \sum _{t=1}^E \frac{d^{\theta ^{i-1}\alpha _0}_t(\mathbf{u}^*) }{ d^{\theta ^i \alpha _0}_t(\mathbf{u}^*) } \le \frac{(1+\rho ) E}{(1-\rho ) \theta }, \end{aligned}$$
where we have used that \( {d^{\theta ^{i-1}\alpha _0}_t(\mathbf{u}^*) }/{ d^{\theta ^i \alpha _0}_t(\mathbf{u}^*) } \le \theta ^{-1}\) by Lemma 5.4. Putting this together with (5.16), we thus obtain \( J_i \lesssim \ln ( d)\), with a uniform constant depending on \(\gamma , \varGamma , \nu \), and \(\theta \). In view of (5.15), this implies that \(\mathbf{u}_k\) converges to \(\mathbf{u}^*\) at a linear rate in cases (5.4) and (5.5).
Step 4 In order to establish rank estimates, we need to bound the errors of \(\mathbf{w}_{i,j}\) as defined in (5.13) for each \(i \ge 0\) and \(0 < j \le J_i\). Since \(\mathbf{u}_0 = \mathbf{u}^{\eta _0} = 0\) by our choice of \(\alpha _0\), for \(i=0\), we obtain
$$\begin{aligned} || \mathbf{w}_{0,j} - \mathbf{u}^* || \le \rho || \mathbf{u}_{0,j-1} - \mathbf{u}^* || = \rho || \mathbf{u}^{\eta _0} - \mathbf{u}^* ||, \quad j = J_0 = 1, \end{aligned}$$
(5.17)
and for \(i>0\) and \(j>0\), by (5.15),
$$\begin{aligned} || \mathbf{w}_{i,j} - \mathbf{u}^* ||\le & {} \rho || \mathbf{u}_{i,j-1} - \mathbf{u}^*|| \nonumber \\\le & {} ( 1 - \nu )^{-1} \rho ^{j} ||\mathbf{u}^{\eta _{i-1}} - \mathbf{u}^*|| + \rho (1 + \rho ^{j-1}) || \mathbf{u}^{\eta _i} - \mathbf{u}^*||. \end{aligned}$$
(5.18)
By Lemma 4.3, with \(M {:}{=} \max _t |\sigma _t(\mathbf{u}^*)|_{\ell ^{p,\infty }}\), for all t and \(0 < j \le J_i\), we have
$$\begin{aligned} {{\mathrm{rank}}}_t (\mathbf{u}_{i,j} ) \lesssim \frac{ || \mathbf{w}_{i,j} - \mathbf{u}^* ||^2 }{ \eta _i^2} + f(\eta _i),\quad f(\eta _i) {:}{=} {\left\{ \begin{array}{ll} M^p \eta _i^{-p}, &{} \text { in case} (5.4),\\ (1 + |\ln \eta _i|)^{\frac{1}{\beta }} ,&{} \text { in case} (5.5). \end{array}\right. } \end{aligned}$$
Note that this also covers \(\mathbf{u}_{i,0}\) for \(i \ge 0\), since \(\mathbf{u}_{i,0} = \mathbf{u}_{i-1,J_{i-1}}\) for \(i>0\) and \(\mathbf{u}_{0,0} = 0\).
In case (5.4), as a consequence of (5.9), (5.15), (5.17), (5.18), with Proposition 3.6(i), we obtain
$$\begin{aligned} ||\mathbf{w}_{i,j} - \mathbf{u}^*|| \; \lesssim \; E M^{\frac{p}{2}} \eta _i^{1 - \frac{p}{2}} \end{aligned}$$
for all respective ij, and consequently
$$\begin{aligned} {{\mathrm{rank}}}_t (\mathbf{u}_{i,j} ) \lesssim E^2 M^p \eta _i^{2-p-2} + M^p \eta _i^{-p} = (1 + E^2) M^p \eta _i^{-p}. \end{aligned}$$
By the same argument, we also have \(||\mathbf{u}_{i,j} - \mathbf{u}^*|| \lesssim E M^{\frac{p}{2}} \eta _i^{1 - \frac{p}{2}}\). Setting \(\varepsilon _{i,j} {:}{=} M^{\frac{p}{2}} \eta _i^{1 - \frac{p}{2}}\), we thus have \( ||\mathbf{u}_{i,j} - \mathbf{u}^*|| \lesssim E \varepsilon _{i,j}\) as well as
$$\begin{aligned} {{\mathrm{rank}}}_t (\mathbf{u}_{i,j} ) \lesssim (1+E^2) M^{\frac{1}{s}} \varepsilon _{i,j}^{-\frac{1}{s}},\quad s = \frac{1}{p} - \frac{1}{2}. \end{aligned}$$
This completes the proof of (5.4). In the case (5.5), Proposition 3.6(ii) yields, expanding \(\eta _i = \theta ^i \alpha _0\),
$$\begin{aligned} ||\mathbf{u}_{i,j} - \mathbf{u}^*||, \; ||\mathbf{w}_{i,j} - \mathbf{u}^*|| \; \lesssim \; E ( 1 + i |\ln \theta | )^{\frac{1}{2\beta }} \theta ^i, \end{aligned}$$
and hence
$$\begin{aligned} {{\mathrm{rank}}}_t (\mathbf{u}_{i,j} ) \lesssim E^2 ( 1 + i |\ln \theta | )^{\frac{1}{\beta }} + ( 1 + i |\ln \theta |)^{\frac{1}{\beta }}. \end{aligned}$$
We choose \(\tilde{\theta }\in (\theta , 1)\) and set \(\varepsilon _{i,j} {:}{=} \tilde{\theta }^i\) to obtain \( ||\mathbf{u}_{i,j} - \mathbf{u}^*|| \lesssim E \varepsilon _{i,j} \) and
$$\begin{aligned} {{\mathrm{rank}}}_t (\mathbf{u}_{i,j} ) \lesssim E^2 \biggl [ \biggl ( 1+ \frac{|\ln \theta |}{ |\ln \tilde{\theta }|} |\ln \tilde{\theta }^i| \biggr ) \biggr ]^{\frac{1}{\beta }} \lesssim E^2 \bigl ( 1+ |\ln \varepsilon _{i,j}| \bigr )^{\frac{1}{\beta }}. \end{aligned}$$
This completes the proof of (5.5). \(\square \)

Remark 5.5

The above algorithm is universal in the sense that it does not require knowledge of the decay of the \(\sigma _t(\mathbf{u}^*)\), but we still obtain the same quasi-optimal rank estimates as with \(\alpha _k\) prescribed a priori as in Theorem 4.4. Note that in (5.5), we have absorbed the additional logarithmic factor that is present in the error bound in (4.9) by comparing to a slightly slower rate of linear convergence as in Remark 4.5, but the estimates are essentially of the same type.

Remark 5.6

For the effective convergence rate \({\tilde{\rho }}\) in the statement of Theorem 5.1, as can be seen from the proof (in particular from the estimates for the \(J_i\)), one has an estimate from above of the form \({\tilde{\rho }} \le {\hat{\rho }}^{\frac{1}{\log d}} < 1\), where \(\hat{\rho }\) does not explicitly depend on d (although it may still depend on d through other quantities such as \(\gamma , \varGamma \)). Consequently, combining this with the statements in (5.4) and (5.5), we generally have to expect that the number of iterations required to ensure \(||\mathbf{u}_k - \mathbf{u}^*|| \le \varepsilon \) scales like \(|\log \hat{\rho }|^{-1} \bigl ( (|\log \varepsilon | + \log d ) \log d\bigr )\). Furthermore, as can be seen from the proof, the estimates for ranks and errors deteriorate only by algebraic factors when \(\gamma \) and \(\varGamma \) vary algebraically in d.

Remark 5.7

Our rank estimates treat all matricizations equally: They take into account only the slowest singular value decay among the matricizations and bound the maximum hierarchical rank. If the low-rank approximability of the matricizations is heterogeneous, that is, if the singular values decay substantially faster for certain matricizations, this does not appear in our bounds. Concerning the asymptotic rank bounds, this has only a minor influence, since the total low-rank approximation error is in fact proportional to the largest singular value tail of a matricization. This can be seen as follows: For nonnegative \(r_t, t=1,\ldots ,E\), we have
$$\begin{aligned} \min _{\begin{array}{c} {{\mathrm{rank}}}_t(\mathbf{v})\le r_t \\ t=1,\ldots ,E \end{array}} ||\mathbf{u}^* - \mathbf{v}|| \le \sqrt{ \sum _{t=1}^E \sum _{j>r_t} |\sigma _{t,j}(\mathbf{u}^*)|^2 } \le \sqrt{E} \max _{t=1,\ldots ,E} \sqrt{\sum _{j>r_t} |\sigma _{t,j}(\mathbf{u}^*)|^2}\nonumber \\ \end{aligned}$$
(5.19)
see [24], as well as
$$\begin{aligned} \sqrt{\sum _{j>r_s} |\sigma _{s,j}(\mathbf{u}^*)|^2} = \min _{{{\mathrm{rank}}}_s(\mathbf{w})\le r_s} ||\mathbf{u}^* - \mathbf{w}|| \le \min _{\begin{array}{c} {{\mathrm{rank}}}_t(\mathbf{v})\le r_t \\ t=1,\ldots ,E \end{array}} ||\mathbf{u}^* - \mathbf{v}|| \end{aligned}$$
for \(s=1,\ldots ,E\), and consequently
$$\begin{aligned} \max _{t=1,\ldots ,E} \sqrt{\sum _{j>r_t} |\sigma _{t,j}(\mathbf{u}^*)|^2} \le \min _{\begin{array}{c} {{\mathrm{rank}}}_t(\mathbf{v})\le r_t \\ t=1,\ldots ,E \end{array}} ||\mathbf{u}^* - \mathbf{v}|| \le \sqrt{E} \max _{t=1,\ldots ,E} \sqrt{\sum _{j>r_t} |\sigma _{t,j}(\mathbf{u}^*)|^2}. \end{aligned}$$
The total error is thus proportional to the largest error in a matricization, up to a factor of at most \(\sqrt{E} \sim \sqrt{d}\). In practice, although this is not captured by the theory, the method can in fact exploit different singular value decays of matricizations, as demonstrated in Sect. 6.2. In principle, it would even be possible to use different thresholding parameters for different matricizations, but it is not clear how to suitably control several such thresholds independently by a posteriori rules.

Remark 5.8

It needs to be noted that our entire construction relies quite strongly on special properties of soft thresholding, especially on its non-expansiveness. In particular, if the soft thresholding operations are replaced by hard thresholding, none of our convergence and complexity results any longer apply. Since a number of worst-case estimates are involved, however, this difference is typically not quite as crucial in practice. Using hard thresholding, the scheme becomes heuristic, but it may still converge and perform well. We illustrate this numerically in Sect. 6.2.

5.1 Inexact Evaluation of Residuals

We finally consider a perturbed version of \({{\mathrm{\textsc {STSolve}}}}\) where residuals are no longer evaluated exactly, but only up to a certain relative error. We assume that for each given \(\mathbf{v}\) and \(\delta > 0\), we can produce \(\mathbf {r}\) such that \(||\mathbf {r} - ({\mathcal {A}}\mathbf{v}- {\mathbf {f}})|| \le \delta \).

We will show below that for our purposes, it suffices to ensure a certain relative error for each \(\mathbf {r}_k\) computed in Algorithm 2, or more precisely, to adjust \(\delta \) for each k such that
$$\begin{aligned} ||\mathbf {r}_k - ({\mathcal {A}}\mathbf{u}_k - {\mathbf {f}})|| \le \min \{ \tau _1 ||\mathbf {r}_k||, \tau _2 \omega ^{-1} ||\mathbf{u}_{k+1} - \mathbf{u}_{k}|| \}, \end{aligned}$$
with suitable \(\tau _1, \tau _2 > 0\). This can be achieved by simply decreasing the value of \(\delta \) and recomputing \(\mathbf {r}_k\) (and the resulting \(\mathbf{u}_{k+1}\)) until
$$\begin{aligned} \delta \le \min \{ \tau _1 ||\mathbf {r}_k||, \tau _2 \omega ^{-1}||\mathbf{u}_{k+1} - \mathbf{u}_{k}|| \} \end{aligned}$$
is satisfied. With such a choice of \(\delta \), we then have in particular
$$\begin{aligned} (1-\tau _1) ||\mathbf {r}_k|| \le ||{\mathcal {A}}\mathbf{u}_k - {\mathbf {f}}|| \le (1+\tau _1) ||\mathbf {r}_k||. \end{aligned}$$
Our scheme can be regarded as an extension of the residual evaluation strategy used in [22] in the context of an adaptive wavelet scheme, where the residual error is controlled relative to the norm of the computed residual. In our algorithm, the error tolerance \(\delta _k\) used for each computed \(\mathbf {r}_k\) is adjusted twice: first in line 13 to ensure the accuracy with respect to \(||\mathbf {r}_k||\), and possibly a second time in line 7 (after incrementing k) to ensure the relative accuracy with respect to \(||\mathbf{u}_{k+1} - \mathbf{u}_k||\).

The residual approximations may in particular involve additional rank reduction operations. However, these serve a different purpose from the truncation by soft thresholding: While the soft thresholding step performs a rank reduction that is large enough to guarantee quasi-optimal ranks, the residual approximation needs to happen with an error small enough to ensure that the computed residual is still sufficiently accurate for reducing the error and for controlling the iteration. In particular, the residual approximations by themselves would not suffice to achieve the given rank bounds.

The analysis of the resulting modified Algorithm 3 follows the same lines as the proof of Theorem 5.1, and we obtain the same statements with modified constants.

Proposition 5.9

The statement of Theorem 5.1 holds also for \({{\mathrm{\textsc {STSolve2}}}}\) given in Algorithm 3.

Proof

We do not restate the full proof, but instead indicate how the required estimates are modified. For a given iterate \(\mathbf{u}_k\), we now denote the exact residual by \({\bar{\mathbf {r}}}_k := {\mathcal {A}}\mathbf{u}_k - {\mathbf {f}}\) and the computed residual by \(\mathbf {r}_k\).

For the moment, we assume that the algorithm does not stop at line 14, and that the loop in line 5 always exits due to the first condition; the opposite cases will be addressed separately later. Under these conditions, we have
$$\begin{aligned} ||\mathbf {r}_k - {\bar{\mathbf {r}}}_k || \le \min \bigl \{ \tau _1 ||\mathbf {r}_k||, \tau _2 \omega ^{-1} ||\mathbf{u}_{k+1} - \mathbf{u}_k|| \bigr \} . \end{aligned}$$
We first consider the error reduction in the iteration with thresholding. Note that
$$\begin{aligned}&|| \mathbf{u}_{k+1} - \mathbf{u}_k || \le || \mathbf{u}_{k + 1} - \mathbf {S}_{\alpha _k}(\mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k) ||\nonumber \\&\quad +\,||\mathbf {S}_{\alpha _k}(\mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k) - \mathbf{u}^{\alpha _k} || + ||\mathbf{u}^{\alpha _k} - \mathbf{u}_k||, \end{aligned}$$
(5.20)
where by non-expansiveness of \(\mathbf {S}_{\alpha _k}\),
$$\begin{aligned} || \mathbf{u}_{k + 1} - \mathbf {S}_{\alpha _k}(\mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k) || \le \omega ||\mathbf {r}_k - {\bar{\mathbf {r}}}_k||, \end{aligned}$$
and thus, since \(\omega ||\mathbf {r}_k - {\bar{\mathbf {r}}}_k|| \le \tau _2 ||\mathbf{u}_{k+1} - \mathbf{u}_{k}||\) by our construction, (5.20) gives
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}_k|| \le \frac{1+\rho }{1- \tau _2} || \mathbf{u}_k - \mathbf{u}^{\alpha _k} ||. \end{aligned}$$
As a consequence,
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k} ||\le & {} || \mathbf {S}_{\alpha _k} (\mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k) - \mathbf{u}^{\alpha _k} ||\nonumber \\&\quad + ||\mathbf {S}_{\alpha _k} ( \mathbf{u}_k - \omega \mathbf {r}_k) - \mathbf {S}_{\alpha _k}(\mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k )|| \nonumber \nonumber \\\le & {} \rho ||\mathbf{u}_k - \mathbf{u}^{\alpha _k}|| + \omega ||\mathbf {r}_k - {\bar{\mathbf {r}}}_k|| \nonumber \nonumber \\\le & {} {\hat{\rho }}(\tau _2) ||\mathbf{u}_k - \mathbf{u}^{\alpha _k}||, \qquad {\hat{\rho }}(\tau _2) := \rho + \frac{(1+\rho )\tau _2}{1-\tau _2}, \end{aligned}$$
(5.21)
where \({\hat{\rho }}(\tau _2) <1\) holds precisely under our assumption \(\tau _2 < \frac{1}{2} (1-\rho )\); in other words, the perturbed fixed-point iteration then has the same contractivity property with a modified constant.
Furthermore, we show next that the validity of the modified condition
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}_{k}|| \le B ||\mathbf {r}_{k+1}||, \quad B := \frac{(1 - \rho )(1-\tau _1) \nu }{ (1+\tau _2) (\rho + (1-\tau _2)^{-1}(1+\rho )\tau _2) \varGamma },\qquad \end{aligned}$$
(5.22)
in Algorithm 3, which replaces (5.6), still implies that the corresponding iterates satisfy (5.12).
To this end, as in the proof of Theorem 5.1, it suffices to show that (5.22) implies \(||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k}|| \le \nu ||\mathbf{u}_{k+1} - \mathbf{u}^*||\). On the one hand, by the construction of \(\mathbf {r}_k\) and the standard error estimate for fixed-point iterations, we have
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k}||\le & {} {\hat{\rho }}(\tau _2) ||\mathbf{u}_k - \mathbf{u}^{\alpha _k}|| \le \frac{{\hat{\rho }}(\tau _2)}{1 - \rho } ||\mathbf {S}_{\alpha _k} (\mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k ) - \mathbf{u}_k ||\\\le & {} \frac{{\hat{\rho }}(\tau _2) (1 + \tau _2)}{1-\rho } ||\mathbf{u}_{k+1} - \mathbf{u}_k||, \end{aligned}$$
and on the other hand, by the construction of \(\mathbf {r}_{k+1}\),
$$\begin{aligned} ||\mathbf{u}_{k+1} - \mathbf{u}^*|| \ge \varGamma ^{-1} ||{\bar{\mathbf {r}}}_{k+1}|| \ge (1- \tau _1) \varGamma ^{-1} ||\mathbf {r}_{k+1}||. \end{aligned}$$
Combining these two estimates, we find that (5.22) implies (5.12).
To carry over the inequalities (5.17) and (5.18), we also need an estimate for the influence of the perturbation on the iteration without thresholding, but with inexact residual, for which we obtain
$$\begin{aligned} || ( \mathbf{u}_k - \omega \mathbf {r}_k ) - \mathbf{u}^* ||\le & {} || (\mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k) - \mathbf{u}^* || + \omega ||\mathbf {r}_k - {\bar{\mathbf {r}}}_k ||\\\le & {} \rho ||\mathbf{u}_k - \mathbf{u}^*|| + \tau _1 \omega ||\mathbf {r}_k||. \end{aligned}$$
Using \(||\mathbf {r}_k || \le (1 - \tau _1)^{-1} ||{\bar{\mathbf {r}}}_k || \le ( 1- \tau _1)^{-1} \varGamma ||\mathbf{u}_k - \mathbf{u}^*||\) and \(\omega \varGamma \le 2\), this gives
$$\begin{aligned} || ( \mathbf{u}_k - \omega \mathbf {r}_k ) - \mathbf{u}^* || \le \biggl ( \rho + \frac{ 2 \tau _1 }{ 1 - \tau _1} \biggr ) ||\mathbf{u}_k - \mathbf{u}^*||. \end{aligned}$$
(5.23)
In order to proceed analogously to the first respective inequalities in (5.17) and (5.18), \(\tau _1 < 1\) thus suffices, since the use of (5.23) is not iterated.

With (5.22) implying (5.12) and the modified estimates (5.21) and (5.23), one can now follow the proof of Theorem 5.1 to obtain the same statements.

Finally, we address the two additional checks in lines 5 and 14 which ensure that the loops for adjusting \(\delta _k\) always terminate. On the one hand, when the condition in line 14 of Algorithm 3 is satisfied, then \(||{\mathcal {A}}\mathbf{u}_{k+1} - {\mathbf {f}}|| \le \gamma \varepsilon \), which implies \(||\mathbf{u}_{k+1}-\mathbf{u}^*|| \le \varepsilon \), and we can therefore terminate the algorithm.

On the other hand, if the loop in line 5 exits because the second condition with the constant
$$\begin{aligned} D := \min \biggl \{ \frac{(1-\tau _1) \tau _2 B}{(1 + \tau _1 + \varGamma B )\omega }, \; \frac{ \rho \nu \tau _2 (1-\tau _1)^2 }{ \bigl ( \rho (1+\tau _1)(1+\tau _2) + \nu (1-\tau _1) (1-\rho ) \bigr ) \omega } \biggr \}\qquad \end{aligned}$$
(5.24)
is violated, that is, if
$$\begin{aligned} \delta _k \le D ||\mathbf {r}_k||, \end{aligned}$$
(5.25)
then the condition \(|| \mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k} || \le \nu ||\mathbf{u}_{k+1} - \mathbf{u}^*||\) as in (5.11) is necessarily satisfied and condition (5.22) in line 18 is guaranteed to hold, which means that \(\alpha _{k}\) will be decreased. To see this, note first that
$$\begin{aligned} ||\mathbf {r}_{k+1}|| \ge (1 + \tau _1)^{-1} \bigl ( (1 - \tau _1) ||\mathbf {r}_k|| - \varGamma ||\mathbf{u}_{k+1} - \mathbf{u}_k || \bigr ). \end{aligned}$$
(5.26)
Since the first condition in line 5 still holds, we have \(||\mathbf{u}_{k+1} - \mathbf{u}_k|| \le \omega \tau _2^{-1} \delta _k\). Therefore, (5.25) implies in particular
$$\begin{aligned} \bigl ( 1+ (1 + \tau _1)^{-1} \varGamma B \bigr ) ||\mathbf{u}_{k+1} - \mathbf{u}_k|| \le B ( 1 + \tau _1)^{-1} (1-\tau _1) ||\mathbf {r}_k||, \end{aligned}$$
which combined with (5.26) implies (5.22).
Furthermore, (5.25) also yields, with the second case in the minimum in (5.24), the estimate
$$\begin{aligned} \frac{\rho }{1 - \rho } (\tau _2^{-1} \omega + \omega ) \delta _k \le \nu \varGamma ^{-1} ( 1 - \tau _1 ) ( 1 + \tau _1)^{-1} \bigl ( (1 - \tau _1) ||\mathbf {r}_k|| - \varGamma \omega \tau _2^{-1} \delta _k \bigr ).\nonumber \\ \end{aligned}$$
(5.27)
Since \(\nu ||\mathbf{u}_{k+1} - \mathbf{u}^*|| \ge \nu \varGamma ^{-1} ||{\bar{\mathbf {r}}}_k || \ge \nu \varGamma ^{-1} (1-\tau _1) ||\mathbf {r}_{k+1}||\), by (5.26) the right-hand side in (5.27) can be estimated from above by \(\nu ||\mathbf{u}_{k+1} - \mathbf{u}^*||\). For the left-hand side, we have
$$\begin{aligned} \frac{\rho }{1 - \rho } (\tau _2^{-1} \omega + \omega ) \delta _k \ge \frac{\rho }{1 - \rho } \bigl ( ||\mathbf{u}_{k+1} - \mathbf{u}_k|| + \omega ||\mathbf {r}_k - {\bar{\mathbf {r}}}_k|| \bigr )\\ \ge \frac{\rho }{1 - \rho } ||\mathbf {S}_{\alpha _k} ( \mathbf{u}_k - \omega {\bar{\mathbf {r}}}_k) - \mathbf{u}_k|| \ge ||\mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k}||, \end{aligned}$$
and from (5.27), altogether we obtain \(|| \mathbf{u}_{k+1} - \mathbf{u}^{\alpha _k} || \le \nu ||\mathbf{u}_{k+1} - \mathbf{u}^*||\) as required. \(\square \)

Remark 5.10

Note that as a consequence of the choice of residual approximation tolerances, the \(\delta _k\) obtained in Algorithm 3 remain proportional to \(||\mathbf {r}_k||\) during the iteration.

5.2 Comparison to Existing Results

As we have noted in Sect. 1.2, complexity estimates for low-rank solvers, including in particular bounds for the arising ranks, have also been obtained in [2, 3]. In these bounds, the low-rank structure of the operator plays a role, in contrast to the bounds obtained here. We now consider these differences in more detail.

The adaptive method in [2, 3] does not address the same problem as we are considering here, since it is designed to also automatically select a discretization for operator equations. One can, however, extract the basic mechanism for controlling ranks. Similarly to the iterative schemes proposed in this work, the resulting procedure \({{\mathrm{\textsc {BD15Solve}}}}\), given in Algorithm 4, can be formulated on infinite-dimensional tensor product Hilbert spaces or on finite-dimensional subspaces. We now compare the rank bounds that one obtains for this method by the results in [2] to those of Theorem 5.1.

Here, we denote by \(\mathbf {T}_\delta \) the standard truncation of the HSVD with \(\ell ^2\)-error bound \(\delta >0\) as introduced in [24], which can be interpreted as a hard thresholding procedure, applied independently to each matricization in the dimension tree and with thresholds adjusted to yield a specified error.
Algorithm 4 has an outer and an inner iteration. For the iterates \(\mathbf{v}_k\) produced in line 11 at the end of each outer iteration, as a consequence of the error reduction achieved in the preceding inner iterations and of [2, Lem. 2], setting
$$\begin{aligned} {\bar{r}}(\mathbf{u}^*,\delta ) := \min \bigl \{ r\in {\mathbb {N}}:\; \bigl (\exists \mathbf{v}\in {\mathcal {H}}, \,\max _t {{\mathrm{rank}}}_t(\mathbf{v})\le r:\; ||\mathbf{u}^*-\mathbf{v}|| \le \delta \bigr ) \bigr \} \end{aligned}$$
and \(\varepsilon _k := \vartheta ^k \varepsilon _0\), one obtains
$$\begin{aligned} ||\mathbf{u}^* - \mathbf{v}_k|| \le \varepsilon _k ,\quad \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{v}_k) \le {\bar{r}}\left( \mathbf{u}^*, \frac{\tau }{1 + \sqrt{E}(1+\tau )} \varepsilon _k\right) . \end{aligned}$$
(5.28)
Under the approximability assumptions of Theorem 5.1, by (5.19), we obtain the following:
  1. (i)
    If \(\sigma _t (\mathbf{u}^*) \in \ell ^{p,\infty }, t=1,\ldots ,E\), with \(0<p<2\), then
    $$\begin{aligned} {\bar{r}}(\mathbf{u}^*,\delta ) \le C_1 d^{\frac{1}{2s}} \max _{t=1,\ldots ,E} |\sigma _t(\mathbf{u}^*)|_{\ell ^{p,\infty }}^{\frac{1}{s}} \delta ^{-\frac{1}{s}} ,\quad s = \frac{1}{p}-\frac{1}{2}, \end{aligned}$$
    (5.29)
    where \(C_1>0\) depends on p and \(\tau \). Hence, (5.28) becomes
    $$\begin{aligned} ||\mathbf{u}^* - \mathbf{v}_k|| \le \varepsilon _k ,\quad \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{v}_k) \lesssim d^{\frac{1}{s}} \max _{t=1,\ldots ,E} |\sigma _t(\mathbf{u}^*)|_{\ell ^{p,\infty }}^{\frac{1}{s}} \varepsilon _k^{-\frac{1}{s}}. \end{aligned}$$
     
  2. (ii)
    If \(\sigma _{t,j}(\mathbf{u}^*) \le C e^{- c j^\beta }\), for \(j\in {\mathbb {N}}\) and \(t=1,\ldots ,E\), with \(C, c, \beta > 0\), then
    $$\begin{aligned} {\bar{r}}(\mathbf{u}^*,\delta ) \le C_2 \,\bigl ( \ln d + |\ln \delta | \bigr )^{\frac{1}{\beta }} , \end{aligned}$$
    (5.30)
    where \(C_2>0\) depends on \(C,c,\beta ,\tau \). In this case, (5.28) yields
    $$\begin{aligned} ||\mathbf{u}^* - \mathbf{v}_k|| \le \varepsilon _k ,\quad \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{v}_k) \lesssim \bigl ( \ln d + |\ln \varepsilon _k| \bigr )^{\frac{1}{\beta }} . \end{aligned}$$
     
Thus, the rank bounds for the iterates \(\mathbf{v}_k\) in the outer iteration obtained after the rank reduction step have the same quasi-optimal dependence on \(\varepsilon _k\) as in Theorem 5.1. Concerning the dependence on d, the bounds for \(\mathbf{v}_k\) do not depend on the properties of \({\mathcal {A}}\) and in fact have a more favorable explicit d-dependence than in Theorem 5.1.

However, the rank bounds obtainable for the iterates \(\mathbf{w}_{k,j}\) of the inner iteration—which of course contribute equally to the computational complexity—depend on \({\mathcal {A}}\) and on the bound I for the number of inner iterations. For discussing this dependence, we need to not only track explicit dependencies on d that arise in the algorithm, but also implicit dependencies due to the problem data. We assume a d-dependent family of comparable problems given by \(({\mathcal {A}},{\mathbf {f}})\). For simplicity, we assume that \({\mathbf {f}}\) is of fixed bounded rank, although our conclusions are valid also when a suitable routine for obtaining low-rank approximations of \({\mathbf {f}}\) is available.

We again obtain estimates of the form
$$\begin{aligned}&||\mathbf{u}^* - \mathbf{w}_{k,j}|| \le \varepsilon _k ,\nonumber \\&\quad \max _{t=1,\ldots ,E} {{\mathrm{rank}}}_t(\mathbf{w}_{k,j}) \lesssim C(d,\varepsilon _k) \, d^{\frac{1}{s}} \max _{t=1,\ldots ,E} |\sigma _t(\mathbf{u}^*)|_{\ell ^{p,\infty }}^{\frac{1}{s}} \varepsilon _k^{-\frac{1}{s}} \end{aligned}$$
(5.31)
in case (i), with an additional factor \(C(d,\varepsilon _k) > 0\); in case (ii), the right side of the second estimate is replaced by \(C(d,\varepsilon _k) \bigl ( \ln d + |\ln \varepsilon _k| \bigr )^{\frac{1}{\beta }}\).

We now consider the particular bounds for \(C(d,\varepsilon _k)\) that one obtains in various situations. It needs to be noted that the resulting worst-case estimates for \(\mathbf{w}_{k,j}\) are typically very pessimistic. In practice, one has additional flexibility in choosing \({\tilde{\kappa }}>0\) to perform additional rank truncations in line 8 of the inner iteration, whose effect is also studied in the numerical examples of Sect. 6, but which do not change the estimates.

We first assume that \({\mathcal {A}}\) has rank bounded by R, that is, one has \({{\mathrm{rank}}}_t({\mathcal {A}}\mathbf{v}) \le R {{\mathrm{rank}}}_t(\mathbf{v})\) for all \(\mathbf{v}\) and t, and that \(\gamma ^{-1}\varGamma \) is bounded independently of d. Then, due to the choice of \(\kappa _1\), we have \(I \le m \ln d\) for some \(m>0\) independent of d. Since we cannot quantify the effect of the additional recompression in line 8, we can only infer
$$\begin{aligned} {{\mathrm{rank}}}_t( \mathbf{w}_{k,j}) \lesssim R^{m \ln d} \bigl (1+ {{\mathrm{rank}}}_t(\mathbf{w}_{k,0}) \bigr ) = d^{m \ln R} \bigl (1+ {{\mathrm{rank}}}_t(\mathbf{w}_{k,0}) \bigr ) . \end{aligned}$$
As a consequence, we obtain (5.31) with \(C(d,\varepsilon _k) = C(d) = d^{m \ln R}\). Already for moderate m and R, the result of Theorem 5.1 for all iterates of \({{\mathrm{\textsc {STSolve}}}}\) (where the polynomial d-dependence is unaffected by R) is thus more favorable than the estimate (5.31) for \({{\mathrm{\textsc {BD15Solve}}}}\) in this case.

If instead, for instance, \(R \sim d^q\) for a \(q>0\), we obtain \(C(d,\varepsilon _k) = C(d) = d^{ q m (\ln d + \ln R)}\), which is superalgebraic in d. If \({\mathcal {A}}\) is not of bounded rank, but can be approximated so that effectively \(R \sim |\ln \varepsilon _k|\), then we instead obtain \(C(d,\varepsilon _k) = |\ln \varepsilon _k|^{m \ln d}\). In such cases, Theorem 5.1 (or Proposition 5.9, respectively) becomes substantially more favorable than (5.31).

The difference becomes more pronounced when the condition number of \({\mathcal {A}}\) is allowed to grow algebraically in d, in which case one obtains \(C(d,\varepsilon _k)\) that has exponential-type growth in d. In contrast, inspecting the proof of Theorem 5.1, one finds that the d-dependence in the rank bounds for \({{\mathrm{\textsc {STSolve}}}}\) and \({{\mathrm{\textsc {STSolve2}}}}\) remains algebraic in d.

The above complexity bounds hold for any nonnegative choice of the parameter \({\tilde{\kappa }}\) in \({{\mathrm{\textsc {BD15Solve}}}}\). Although the effect of the additional truncations for positive \({\tilde{\kappa }}\) is not rigorously quantified, they typically lead to ranks that are substantially smaller than the theoretical bounds for \(\mathbf{w}_{k,j}\), and in this regard, \({{\mathrm{\textsc {BD15Solve}}}}\) offers some additional flexibility. This will be illustrated in more detail in the following section.

6 Numerical Experiments

The procedures \({{\mathrm{\textsc {STSolve}}}}\) and \({{\mathrm{\textsc {STSolve2}}}}\) can be applied to quite general discretized elliptic problems, since only bounds on the spectrum of the discrete operator \({\mathcal {A}}\) are required. For our numerical tests2, we first consider a high-dimensional Poisson problem, including a comparison to the results obtained with \({{\mathrm{\textsc {BD15Solve}}}}\) as described in Sect. 5.2, as well as a parametric diffusion problem.

In our tests, we focus on the evolution of the ranks of iterates, using discretizations such that the discretization errors remain negligible in the considered tests. Depending on how precisely the residual approximation is realized in \({{\mathrm{\textsc {STSolve2}}}}\), complexity bounds for the iteration may be dominated by those of the soft thresholding procedure, as discussed in Remark 3.1.

6.1 A High-Dimensional Poisson Problem

As a first test, we consider application of Algorithm 3 to a discretized Poisson problem with homogeneous Dirichlet boundary conditions,
$$\begin{aligned} -\varDelta u = f \quad \text {on}(0,1)^d, \end{aligned}$$
using similar techniques based on wavelet discretization as for the adaptive treatment in [3]. In this setting, we have a suitable method for preconditioning with explicit control of the resulting condition numbers.

In principle, for treating such partial differential equations, one needs to adjust the discretization to the desired solution accuracy, as analyzed in detail in [3]. For our present simplified test, we fix a sufficiently fine discretization, corresponding to a fixed choice of wavelet basis functions, in advance. Based on the analysis in [3], we then choose a corresponding preconditioner that guarantees a certain uniform bound on the condition number for this particular discretization.

We shall now briefly describe how the discrete operator \({\mathcal {A}}\) is obtained as a symmetrically preconditioned Galerkin discretization in a tensor product wavelet basis. Starting from an orthonormal basis of sufficiently regular (multi-)wavelets \(\{ \psi _\nu \}_{\nu \in \nabla }\) of \(L^2(0,1)\), we obtain a tensor product orthonormal basis \(\{ \varPsi _\nu \}_{\nu \in \nabla ^d}\) of \(L^2((0,1)^d)\) with \(\varPsi _\nu := \psi _{\nu _1}\otimes \cdots \otimes \psi _{\nu _d}\), such that the rescaled basis functions \(c_\nu ^{-1} \varPsi _\nu , \nu \in \nabla ^d\), where
$$\begin{aligned} c_\nu := \Bigl ( ||\psi _{\nu _1}||^2_{H^1_0(0,1)} + \cdots + ||\psi _{\nu _d}||^2_{H^1_0(0,1)} \Bigr )^{\frac{1}{2}}, \end{aligned}$$
form a Riesz basis of \(H^1_0((0,1)^d)\). We now pick a fixed finite subset \(\varLambda _1 \subset \nabla \) and set \(\varLambda := \varLambda _1 \times \cdots \times \varLambda _1 \subset \nabla ^d\). Furthermore, we use the family of low-rank approximate diagonal scaling operators \({\tilde{\mathbf { S}}}_n^{-1}, n\in {\mathbb {N}}\), constructed in [3]: We choose a \({\bar{\delta }}\in (0,1)\) and then take \(\bar{n}\) according to [3, Theorem 4.1] such that
$$\begin{aligned} (1- {\bar{\delta }}) \bigl ||{{\mathrm{diag}}}(c_\nu ^{-1}) \mathbf{v}\bigr || \le \bigl ||{\tilde{\mathbf { S}}}^{-1}_{\bar{n}} \mathbf{v}\bigr || \le (1+{\bar{\delta }}) \bigl ||{{\mathrm{diag}}}(c_\nu ^{-1}) \mathbf{v}\bigr || \end{aligned}$$
for all sequences \(\mathbf{v}\) supported on \(\varLambda \); note that the value of \(\bar{n}\), which corresponds to the rank of \({\tilde{\mathbf { S}}}^{-1}_{\bar{n}}\), depends in particular on \(\varLambda \). With
$$\begin{aligned} \mathbf {T}_\varLambda := \left( \sum _{i=1}^d \langle \partial _i \varPsi _\nu ,\partial _i \varPsi _\mu \rangle _{L^2} \right) _{\lambda ,\nu \in \varLambda },\quad \mathbf { g}_\varLambda := \bigl ( \langle f, \varPsi _\nu \rangle \bigr )_{\nu \in \varLambda }, \end{aligned}$$
we then set
$$\begin{aligned} {\mathcal {A}}:= {\tilde{\mathbf { S}}}^{-1}_{\bar{n}} \mathbf { T}_\varLambda {\tilde{\mathbf { S}}}^{-1}_{\bar{n}},\quad {\mathbf {f}} := {\tilde{\mathbf { S}}}^{-1}_{\bar{n}} \mathbf {g}_\varLambda . \end{aligned}$$
Thus, \(\mathbf{u}^* = {\tilde{\mathbf { S}}}_{\bar{n}} \mathbf { T}_\varLambda ^{-1}\mathbf {g}_\varLambda \), where the additional scaling by \({\tilde{\mathbf { S}}}_{\bar{n}}\) yields convergence of the scheme in \(H^1\)-norm at a controlled rate. An approximation of \(\mathbf { T}_\varLambda ^{-1}\mathbf {g}_\varLambda \), which in turn is a Galerkin approximation of the sequence of \(L^2\)-coefficients \(\langle u,\varPsi _\nu \rangle \) of the true solution, can then be recovered by applying \({\tilde{\mathbf { S}}}^{-1}_{\bar{n}}\) to the computed \(\mathbf{u}_\varepsilon \). For \({\mathcal {A}}\), one can obtain accurate bounds for \(\gamma , \varGamma \), and in particular,
$$\begin{aligned} \kappa \le \frac{(1+{\bar{\delta }})^2}{(1-{\bar{\delta }})^2} {\text {cond}}_2 \bigl ( c_\lambda ^{-1} \langle \psi _\lambda ', \psi _\nu '\rangle c_\nu ^{-1} \bigr )_{\lambda ,\nu \in \varLambda _1}. \end{aligned}$$
In our numerical tests, as in [3], we take \(f = 1\) and use the piecewise polynomial, continuously differentiable orthonormal multiwavelets of order 7 constructed in [19]. The univariate index set \(\varLambda _1\) comprises all multiwavelet basis functions on levels \(0,\ldots , 4\), which yields \(\#(\varLambda _1) = 224\). The parameters in Algorithm 3 are chosen as \(\theta := \frac{3}{4}, \nu := \frac{9}{10}, \alpha _0 := \frac{1}{2} \omega ||{\mathbf {f}}||, \tau _1 = \frac{1-\rho }{2(3-\rho )}, \tau _2=\frac{1}{4}(1-\rho )\), and for the preconditioner, we take \({\bar{\delta }} := \frac{1}{10}\).

Note that since the resulting diagonal scalings \({\tilde{\mathbf { S}}}^{-1}_{\bar{n}}\) consist of 10 separable terms (whose number increases with larger \(\varLambda _1\)), a naive direct application of \({\mathcal {A}}\) could increase the hierarchical ranks of a given input \(\mathbf{v}\) by a factor of up to 200; the observed ranks required for accurately approximating \({\mathcal {A}}\mathbf{v}\), however, are much lower. Therefore, we use the recompression strategy described in [3, Section 7.2] for an approximate evaluation of \({\mathcal {A}}\mathbf{v}\) with prescribed tolerance in order to avoid unnecessarily large ranks in intermediate quantities. In this setting, the inexact residual evaluation in Algorithm 3 is thus of crucial practical importance.

We compare the computed solutions to a very accurate reference solution of the discrete problem obtained by an exponential sum approximation \({\hat{\mathbf {u}}}_0 \approx \mathbf {T}_\varLambda ^{-1}\mathbf {g}_\varLambda \), see [23, 26]. The error to the reference solution is computed as \(\text {err}_k := ||{{\mathrm{diag}}}(c_\nu ) ({\tilde{\mathbf {S}}}^{-1}_{\bar{n}} \mathbf{u}_k - {\hat{\mathbf {u}}}_0)||\), which is proportional to the error in \(H^1\)-norm of the corresponding represented functions. The quantity \(\text {err}_k\) thus serves as a substitute for the difference in the relevant norm of \(\mathbf{u}_k\) to the exact solution of the discretized problem.
Fig. 2

\({{\mathrm{\textsc {STSolve2}}}}\), example of Sect. 6.1, from left to right (vs. iteration number k): computed discrete residual norms \(||\mathbf {r}_k||\), ratios of differences to reference solution \(\text {err}_k\) to \(||\mathbf {r}_k||\), corresponding thresholding parameters \(\alpha _k\); each for \(d=16\) (light gray), \(d=32\) (dark gray), \(d=64\) (black)

Fig. 3

\({{\mathrm{\textsc {STSolve2}}}}\), example of Sect. 6.1; solid lines maximum and minimum hierarchical ranks of iterates, i.e., \(\min _t {{\mathrm{rank}}}_t(\mathbf{u}_k)\) and \(\max _t {{\mathrm{rank}}}_t(\mathbf{u}_k)\); dotted lines maximum ranks of computed residuals, \(\max _t {{\mathrm{rank}}}_t(\mathbf {r}_k)\); each versus k

The numerical results for \(d=16, d=32\), and \(d=64\) are shown in Figs. 2 and 3. In each case, we use the linear dimension tree (2.1). It can be observed in Fig. 2 that the norm of the solution of the problem decreases slightly with increasing d; apart from this, the iteration behaves very similarly for the different values of d, producing in particular a monotonic decrease in discrete residual norms. As expected, these values also remain uniformly proportional to the \(H^1\)-difference to the reference solution. The values \(\alpha _k\) can be seen to first decrease in every step as long as \(\mathbf{u}_k = 0\). Subsequently, they decrease in a regular manner after an essentially constant number of iterations. As one would also expect, the final value of \(\alpha _k\) needs to be slightly smaller for larger d.

Figure 3 shows the maximum and minimum hierarchical ranks of the computed iterates (whose difference grows slightly with increasing d) compared to the ranks of the corresponding computed discrete residuals \(\mathbf {r}_k\), clearly demonstrating the reduced rank increase relative to \(\mathbf{u}_k\) that we obtain by the approximate residual evaluation. The additional variation in the residual ranks is a consequence of the fact that the differences \(||\mathbf{u}_{k+1} - \mathbf{u}_k||\) decrease as long as \(\alpha _k\) remains constant, enforcing a more accurate residual evaluation. As soon as the thresholding parameter changes, the accuracy requirement is subsequently relaxed again by line 20 in Algorithm 3, since the values \(||\mathbf{u}_{k+1} - \mathbf{u}_k||\) again become larger when \(\mathbf{u}^{\alpha _k}\) changes. Note furthermore that the ranks show little variation with increasing d, which is substantially more favorable than the quadratic increase with d that is possible in the estimates of Theorem 5.1.
Fig. 4

\({{\mathrm{\textsc {BD15Solve}}}}\), example of Sect. 6.1 with \(d=16\); left residual bound \(||\mathbf {r}_{k,j}||+\eta _{k,j}\) versus iteration count; right ranks of \(\mathbf{w}_{k,j}\) (open square) and \(\mathbf {r}_{k,j}\) (cross) versus iteration count, where filled squares mark outer loop iterates \(\mathbf{v}_k=\mathbf{w}_{k,0}\)

Fig. 5

Example of Sect. 6.1, \(d=16\); left maximum ranks of \(\mathbf{u}_k\) in \({{\mathrm{\textsc {STSolve2}}}}\) (filled circle) and \(\mathbf{w}_{k,j}\) in \({{\mathrm{\textsc {BD15Solve}}}}\) (open square), right maximum ranks of \(\mathbf {r}_k\) in \({{\mathrm{\textsc {STSolve2}}}}\) (open circle) and \(\mathbf {r}_{k,j}\) in \({{\mathrm{\textsc {BD15Solve}}}}\) (cross), each versus the achieved residual bound. Filled squares mark ranks of outer loop iterates \(\mathbf{v}_k=\mathbf{w}_{k,0}\)

Table 1

Comparison of execution times (in seconds) for ensuring a residual norm below \(10^{-3}\), on a machine with 2.5 GHz Core i7-4870HQ CPU, for the example of Sect. 6.1 with \(d=16\)

\({{\mathrm{\textsc {STSolve2}}}}\)

\(\nu \)

\(\theta \)

0.01

0.025

0.1

0.4

   0.9

218.6

86.2

77.9

175.1

   0.3

227.6

95.9

90.1

244.4

\({{\mathrm{\textsc {BD15Solve}}}}\)

\(\vartheta \)

\({\tilde{\kappa }}\)

0.01

0.1

1

100

   0.5

2233.3

27.6

11.7

23.6

   0.1

1490.9

16.4

6.7

13.0

We also compare to the results obtained by \({{\mathrm{\textsc {BD15Solve}}}}\) as in Algorithm 4, using the same procedure for approximating residuals. As noted in Sect. 5.2, the rank bounds for the intermediate iterates \(\mathbf{w}_{k,j}\) are in general weaker than those for the iterates of \({{\mathrm{\textsc {STSolve2}}}}\), but one has additional flexibility in choosing the parameter \({\tilde{\kappa }}\) that controls the additional rank truncations in the inner iteration. The results for \(d=16\), with \({\tilde{\kappa }}=\frac{1}{10}\) and \(\vartheta =\frac{1}{2}\), are given in Fig. 4 and compared to those of \({{\mathrm{\textsc {STSolve2}}}}\) in Fig. 5. Here, the intermittent increases in the residuals observed for \({{\mathrm{\textsc {BD15Solve}}}}\) in Fig. 4 are caused by the rank truncations that restore near-optimal ranks at the end of each outer iteration. As can be seen in Fig. 5, the ranks of intermediate iterates are indeed slightly larger than those produced by \({{\mathrm{\textsc {STSolve2}}}}\) at the same residual bounds. Although the residual approximation tolerances used by both methods remain proportional to the current solution error, the residual tolerances used by \({{\mathrm{\textsc {STSolve2}}}}\) are lower by a certain factor, leading to larger ranks of the resulting approximate residuals \(\mathbf {r}_k\).

The latter difference also plays a role in our comparison of effective computational complexities in Table 1. Here, we juxtapose several choices of parameters in the algorithms, in each case within the ranges where the respective theoretical complexity estimates apply. Mainly as a consequence of the more economical residual approximation tolerances, \({{\mathrm{\textsc {BD15Solve}}}}\) can be substantially more efficient than \({{\mathrm{\textsc {STSolve2}}}}\). However, this depends strongly on a sufficiently large choice of \({\tilde{\kappa }}\), which enables \({{\mathrm{\textsc {BD15Solve}}}}\) to profit from rank truncations beyond the theoretical guarantees. For small \({\tilde{\kappa }}\), where one approaches the more pessimistic rank bounds ensured by the analysis, this advantage is evidently lost. In contrast, \({{\mathrm{\textsc {STSolve2}}}}\) shows fairly small variations in performance for the various choices of parameters.

6.2 A Parametric Diffusion Problem

We now turn to a second test problem, an elliptic differential equation with diffusion coefficient depending on countably many parameters. That is, with \(U:=[-1,1]^{{\mathbb {N}}}\), we consider the parametrized coefficient functions
$$\begin{aligned} a(y) := 1 + \sum _{j \ge 1} y_j \psi _j , \quad y \in U, \end{aligned}$$
(6.1)
where \(\psi _j(x) :=c_0 j^{-\frac{7}{2}} \sin (\pi j x)\) with \(c_0 := (2\zeta (\frac{7}{2}))^{-1}\), and the problem of finding \(u\in L^2(U,V,\mu ) \simeq V\otimes L^2(U,\mu )\), with \(V:= H^1_0(0,1)\) and \(\mu \) the uniform measure on U, such that \(u(y)\in V\) for each \(y\in U\) solves
$$\begin{aligned} - \bigl ( a(y) \,u(y) \bigr )' = 1, \quad u(0)=u(1) = 0 \end{aligned}$$
in the weak sense, which yields an elliptic operator equation on \(L^2(U,V,\mu )\). To arrive at a finite-dimensional problem, we truncate the summation in (6.1) to \(j=1,\ldots ,J\) for some J and introduce finite-dimensional subspaces \(V_h\subset V\) and \(W_L \subset L^2((-1,1),\frac{1}{2} d\lambda )\). Specifically, for \(V_h\), we take piecewise quadratic finite elements of uniform grid size h, and \(W_L := {\text {span}} \{ L_k \}_{k=0,\ldots ,L}\), where \(L_k\) are the orthonormal Legendre polynomials. We thus arrive at
$$\begin{aligned} {\mathcal {A}}_0 = {\bar{\mathbf {A}}} \otimes \mathrm {Id}+ \sum _{j=1}^J \mathbf {A}_j \otimes \mathbf {M}_j \end{aligned}$$
where \({\bar{\mathbf {A}}}\) and \(\mathbf {A}_j\) are the discretizations of the diffusion operator with coefficients 1 and \( \psi _j\), respectively, and \(\mathbf {M}_j\) is the discretization of multiplication by \(y_j\) on \(W_L\). For preconditioning, we use a sparse Cholesky factorization \({\bar{\mathbf {A}}} = \mathbf {L}\mathbf {L}^T\), to arrive at
$$\begin{aligned} {\mathcal {A}}:= (\mathbf {L}^{-1}\otimes \mathrm {Id}) {\mathcal {A}}_0 (\mathbf {L}^{-T}\otimes \mathrm {Id}) \end{aligned}$$
on \({\mathcal {H}}= {\mathbb {R}}^{\dim V_h} \otimes \bigl (\bigotimes _{j=1}^{J}{\mathbb {R}}^{L+1}\bigr )\). One finds that (5.1) holds with \(\gamma = \frac{1}{2}, \varGamma =\frac{3}{2}\), and therefore (5.2) with \(\omega = 1\) and \(\rho = \frac{1}{2}\).

The characteristics of this problem with respect to low-rank approximation are quite different from those of the discretized Poisson equation. In the latter case, one obtains similar exponential-type decay for the singular values of all matricizations. Here, one instead observes much slower algebraic decay of the singular values of the matricization \({\mathcal {\hat{M}}}_{\{1\}}(\mathbf{u}^*)\) as \(J\rightarrow \infty \), but exponential decay of those of \({\mathcal {\hat{M}}}_{\{i\}}(\mathbf{u}^*)\) for \(i>1\); note that \({\mathcal {\hat{M}}}_{\{1\}}(\mathbf{u}^*)\) plays a special role, since it corresponds to the separation between spatial and stochastic variables.

The present example thus concerns on the one hand the estimates for algebraic decay in Theorem 5.1 and on the other hand the behavior of the iteration for such heterogeneous decay as noted in Remark 5.7. As a consequence of the results in [12], the sequence of singular values of \({\mathcal {\hat{M}}}_{\{1\}}(\mathbf{u}^*)\) is in \(\ell ^{p,\infty }\) for \(p=\frac{2}{7}\), corresponding to algebraic convergence with rate \(s=3\) in Theorem 5.1.

In our tests, we use again the linear dimension tree (2.1). We choose sufficiently large J and L, as well as sufficiently fine spatial discretizations, such that these discretizations do not influence the low-rank approximability at the considered solution residual sizes. The results thus reflect the behavior of the iteration on the full function space, where \({\mathcal {A}}\) has unbounded hierarchical rank.

In Fig. 6, we compare the results for \({{\mathrm{\textsc {STSolve2}}}}\) and \({{\mathrm{\textsc {BD15Solve}}}}\). Both yield a growth of maximum ranks at a rate consistent with the above value of s, but \({{\mathrm{\textsc {STSolve2}}}}\) leads to quantitatively larger ranks. This appears to be related to the larger truncation error incurred by soft thresholding. To illustrate this, we additionally consider a modification of \({{\mathrm{\textsc {STSolve2}}}}\) obtained by replacing all soft thresholding operations by hard thresholding with the same threshold levels, which leads to a purely heuristic method whose convergence is not ensured. Here, this modified scheme performs well, however, and indeed yields even smaller ranks than \({{\mathrm{\textsc {BD15Solve}}}}\). Thus, in this example, the general complexity bounds for soft thresholding come at the price of larger ranks than obtained with less conservative thresholding procedures.

As noted in Remark 5.7, our rank bounds do not take into account the qualitatively different behavior of the singular values of different matricizations. On the one hand, the results in Fig. 6 show that even though in the present case the slowest singular value decay occurs only in a single matricization, it still dominates the convergence behavior. On the other hand, as illustrated by the results in Table 2, \({{\mathrm{\textsc {STSolve2}}}}\) can in practice exploit the faster singular value decay of the other matricizations, even though this is not captured by the theoretical rank bounds.
Fig. 6

Ranks of iterates for the example of Sect. 6.2, open circle\({{\mathrm{\textsc {STSolve2}}}}\), open triangle\({{\mathrm{\textsc {STSolve2}}}}\) with soft thresholding replaced by hard thresholding, open square\({{\mathrm{\textsc {BD15Solve}}}}\), where  filled squares mark outer loop iterates \(\mathbf{v}_k = \mathbf{w}_{k,0}\). The line has slope \(\frac{1}{3}\)

Table 2

Residual bounds and matricization ranks of iterates \(\mathbf{u}_k\) of \({{\mathrm{\textsc {STSolve2}}}}\) obtained for the example of Sect. 6.2

k

\(||\mathbf {r}_k||+\delta _k\)

\({{\mathrm{rank}}}({\mathcal {\hat{M}}}_{\{i\}}(\mathbf{u}_k))\) for \(i=1,\ldots , 15\)

6

\(1.15\text {e}_{-1}\)

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

9

\(3.04\text {e}_{-2}\)

4

3

2

2

1

1

1

1

1

1

1

1

1

1

1

12

\(7.50\text {e}_{-3}\)

9

3

2

2

2

1

1

1

1

1

1

1

1

1

1

15

\(1.92\text {e}_{-3}\)

12

4

3

2

2

2

2

1

1

1

1

1

1

1

1

18

\(4.81\text {e}_{-4}\)

18

5

3

2

2

2

2

2

2

2

1

1

1

1

1

21

\(1.21\text {e}_{-4}\)

26

5

3

3

2

2

2

2

2

2

2

2

2

2

1

7 Conclusion

We have constructed an iterative scheme for solving linear elliptic operator equations in hierarchical tensor representations. This method guarantees linear convergence to the solution \(\mathbf{u}^*\) as well as quasi-optimality of the tensor ranks of all iterates and is universal in the sense that no a priori knowledge on the low-rank approximability of \(\mathbf{u}^*\) is required.

Since the given choices of thresholding parameters work for quite general contractive fixed-point mappings, the construction of schemes that make this choice a posteriori may be possible for more general cases than the linear elliptic one treated here. In this regard, note that although we have always assumed for ease of presentation that the considered operator \({\mathcal {A}}\) is also symmetric, this is not essential.

In this work, we have considered fixed discretizations, and we have obtained bounds that are robust with respect to these discretizations. Although such rank bounds constitute the central ingredient in the complexity analysis of such methods, to obtain meaningful estimates for the total complexity of solving operator equations, this still needs to be combined with adapted discretizations as in [2], which will be the subject of a forthcoming work.

We have seen that our approach generally yields stronger asymptotic rank bounds for each iterate than previous constructions. It also exhibits the expected robust behavior in practice, where it can, however, be quantitatively somewhat more expensive than existing approaches due to the comparably conservative control of the iteration. A further interesting question is therefore to what extent the quantitative performance can be enhanced, for instance by a modified soft thresholding procedure or by a different adjustment of error tolerances, while retaining the asymptotic bounds.

However, our analysis can also be interpreted as a rigorous justification of heuristic low-rank solvers that are based on similar operations. For instance, as can be seen in the test of Sect. 6.2, replacing soft thresholding by hard thresholding means that the general estimates no longer apply, but it may amount only to a small (or even favorable) perturbation in typical practical examples. The methods presented here can thus also serve as a reference point for heuristic schemes that are optimized for particular problems.

Footnotes

  1. 1.

    Alternative notions of nuclear norms of tensors can be defined by duality [53], but these are more difficult to handle.

  2. 2.

    All methods were implemented in C++, using LAPACK for numerical linear algebra operations.

Notes

Acknowledgments

This research was supported in part by DFG SPP 1324 and ERC AdG BREAD.

References

  1. 1.
    Bachmayr, M., Cohen, A.: Kolmogorov widths and low-rank approximations of parametric elliptic PDEs. Math. Comp. (2016). In press.Google Scholar
  2. 2.
    Bachmayr, M., Dahmen, W.: Adaptive near-optimal rank tensor approximation for high-dimensional operator equations. Found. Comput. Math. 15(4), 839–898 (2015)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Bachmayr, M., Dahmen, W.: Adaptive low-rank methods: Problems on Sobolev spaces. SIAM J. Numer. Anal. (2016). In press.Google Scholar
  4. 4.
    Ballani, J., Grasedyck, L.: A projection method to solve linear systems in tensor format. Numer. Linear Algebra Appl. 20(1), 27–43 (2013)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sciences 2, 182–202 (2009)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Beylkin, G., Mohlenkamp, M. J.: Algorithms for numerical analysis in high dimensions. SIAM J. Sci. Comput. 26(6), 2133–2159 (electronic) (2005)Google Scholar
  7. 7.
    Billaud-Friess, M., Nouy, A., Zahm, O.: A tensor approximation method based on ideal minimal residual formulations for the solution of high-dimensional problems. ESAIM: M2AN 48(6), 1777–1806 (2014).Google Scholar
  8. 8.
    Bredies, K., Lorenz, D. A.: Linear convergence of iterative soft-thresholding. J Fourier Anal Appl 14, 813–837 (2008)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Cai, J.-F., Candès, E. J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J Optimization 20, 1956–1982 (2010)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Cancès, E., Ehrlacher, V., Lelièvre, T.: Convergence of a greedy algorithm for high-dimensional convex nonlinear problems. Math. Models Methods Appl. Sci. 21(12), 2433–2467 (2011)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Cohen, A., Dahmen, W., DeVore, R.: Adaptive wavelet methods for elliptic operator equations: Convergence rates. Mathematics of Computation 70(233), 27–75 (2001)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Cohen, A., DeVore, R., Schwab, C.: Analytic regularity and polynomial approximation of parametric and stochastic PDE’s. Analysis and Applications 9, 1–37 (2011)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Da Silva, C., Herrmann, F. J.: Optimization on the hierarchical Tucker manifold—applications to tensor completion. Linear Algebra Appl. 481, 131–173 (2015)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Dahlke, S., Fornasier, M., Raasch, T.: Multilevel preconditioning and adaptive sparse solution of inverse problems. Math. Comp. 81, 419–446 (2012)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Dahmen, W., DeVore, R., Grasedyck, L., Süli, E.: Tensor-sparsity of solutions to high-dimensional elliptic partial differential equations. Found. Comput. Math. (2015). In press. DOI:10.1007/s10208-015-9265-9
  16. 16.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. Pure Appl. Math. 57(11), 1413–1457 (2004)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    DeVore, R.: Nonlinear approximation. Acta Numer. 7, 51–150 (1998)CrossRefMATHGoogle Scholar
  18. 18.
    Dolgov, S. V., Savostyanov, D. V.: Alternating minimal energy methods for linear systems in higher dimensions. SIAM J. Sci. Comput. 36(5), A2248–A2271 (2014)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Donovan, G. C., Geronimo, J. S., Hardin, D. P.: Orthogonal polynomials and the construction of piecewise polynomial smooth wavelets. SIAM J. Math. Anal. 30, 1029–1056 (1999)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Falcó, A., Nouy, A.: Proper generalized decomposition for nonlinear convex problems in tensor Banach spaces. Numer. Math. 121(3), 503–530 (2012)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Gandy, S., Recht, B., Yamada, I.: Tensor completion and low-n-rank tensor recovery via convex optimization. Inverse Problems 27(2), 025010 (2011)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Gantumur, T., Harbrecht, H., Stevenson, R.: An optimal adaptive wavelet method without coarsening of the iterands. Math. Comp. 76(258), 615–629 (2007)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Grasedyck, L.: Existence and computation of low Kronecker-rank approximations for large linear systems of tensor product structure. Computing 72, 247–265 (2004)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Grasedyck, L.: Hierarchical singular value decomposition of tensors. SIAM J. Matrix Anal. Appl. 31(4), 2029–2054 (2010)MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitt. 36(1), 53–78 (2013)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Hackbusch, W.: Entwicklungen nach Exponentialsummen. Tech. Rep. 4, MPI Leipzig (2005)Google Scholar
  27. 27.
    Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus, Springer Series in Computational Mathematics, vol. 42. Springer-Verlag Berlin Heidelberg (2012)CrossRefMATHGoogle Scholar
  28. 28.
    Hackbusch, W.: Numerical tensor calculus. Acta Numer. 23, 651–742 (2014)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Hackbusch, W., Khoromskij, B., Tyrtyshnikov, E.: Approximate iterations for structured matrices. Numerische Mathematik 109, 119–156 (2008)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Hackbusch, W., Kühn, S.: A new scheme for the tensor representation. J. Fourier Anal. Appl. 15(5), 706–722 (2009)MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Holtz, S., Rohwedder, T., Schneider, R.: The alternating linear scheme for tensor optimization in the tensor train format. SIAM J. Sci. Comput. 34(2), A683–A713 (2012)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Huang, B., Mu, C., Goldfarb, D., Wright, J.: Provable models for robust low-rank tensor completion. Pacific Journal of Optimization 11(2), 339–364 (2015)MathSciNetMATHGoogle Scholar
  33. 33.
    Khoromskij, B. N.: Tensor-structured preconditioners and approximate inverse of elliptic operators in \({\mathbb{R}^{d}}\). Constr. Approx. 30(3), 599–620 (2009)MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Khoromskij, B. N.: Tensor numerical methods for multidimensional: PDEs theoretical analysis and initial applications. ESAIM: ProcS 48, 1–28 (2015)MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Khoromskij, B. N., Schwab, C.: Tensor-structured Galerkin approximation of parametric and stochastic elliptic PDEs. SIAM J. Sci. Comput. 33(1), 364–385 (2011)MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Kressner, D., Steinlechner, M., Vandereycken, B.: Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure. preprint (2015)Google Scholar
  37. 37.
    Kressner, D., Tobler, C.: Low-rank tensor Krylov subspace methods for parametrized linear systems. SIAM J. Matrix Anal. Appl. 32, 1288–1316 (2011)MathSciNetCrossRefMATHGoogle Scholar
  38. 38.
    Kressner, D., Uschmajew, A.: On low-rank approximability of solutions to high-dimensional operator equations and eigenvalue problems. Linear Algebra Appl. 493, 556–572 (2016)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Kühn, S.: Hierarchische Tensordarstellung. Ph.D. thesis, Universität Leipzig (2012)Google Scholar
  40. 40.
    Lai, M.-J., Yin, W.: Augmented \(\ell _1\) and nuclear-norm models with a globally linearly convergent algorithm. SIAM Journal on Imaging Sciences 6(2), 1059–1091 (2013)MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion for estimating missing values in visual data. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 208–220 (2013)CrossRefGoogle Scholar
  42. 42.
    Ma, S., Goldfarb, D., Chen, L.: Fixed point and Bregman iterative methods for matrix rank minimization. Mathematical Programming 128(1-2), 321–353 (2011)MathSciNetCrossRefMATHGoogle Scholar
  43. 43.
    Markus, A. S.: The eigen- and singular values of the sum and product of linear operators. Russian Mathematical Surveys 19(4), 91–120 (1964)CrossRefMATHGoogle Scholar
  44. 44.
    Mirsky, L.: Symmetric gauge functions and unitarily invariant norms. Quarterly Journal of Mathematics 11, 50–59 (1960)MathSciNetCrossRefMATHGoogle Scholar
  45. 45.
    Moreau, J.: Proximité et dualité dans un espace hilbertien. Bulletin de la Société Mathématique de France 93, 273–299 (1965)MathSciNetCrossRefMATHGoogle Scholar
  46. 46.
    Oseledets, I., Tyrtyshnikov, E.: Breaking the curse of dimensionality, or how to use SVD in many dimensions. SIAM J. Sci. Comput. 31(5), 3744–3759 (2009)MathSciNetCrossRefMATHGoogle Scholar
  47. 47.
    Oseledets, I. V.: Tensor-train decomposition. SIAM J. Sci. Comput. 33(5), 2295–2317 (2011)MathSciNetCrossRefMATHGoogle Scholar
  48. 48.
    Recht, B., Fazel, M., Parrilo, P. A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review 52(3), 471–501 (2010)MathSciNetCrossRefMATHGoogle Scholar
  49. 49.
    Schneider, R., Uschmajew, A.: Approximation rates for the hierarchical tensor format in periodic Sobolev spaces. J. Complexity 30(2), 56–71 (2014)MathSciNetCrossRefMATHGoogle Scholar
  50. 50.
    Vidal, G.: Efficient classical simulation of slightly entangled quantum computations. Phys. Rev. Lett. 91, 147902 (2003)CrossRefGoogle Scholar
  51. 51.
    White, S. R.: Density matrix formulation for quantum renormalization groups. Phys. Rev. Lett. 69, 2863–2866 (1992)CrossRefGoogle Scholar
  52. 52.
    Wright, S. J., Nowak, R. D., Figueiredo, M. A. T.: Sparse reconstruction by separable approximation. IEEE Trans Sig. Process. 57, 2479–2493 (2009)MathSciNetCrossRefGoogle Scholar
  53. 53.
    Yuan, M., Zhang, C.-H.: On tensor completion via nuclear norm minimization. Found. Comput. Math. (2015). In press. DOI:10.1007/s10208-015-9269-5

Copyright information

© SFoCM 2016

Authors and Affiliations

  1. 1.Sorbonne Universités, UPMC Univ Paris 06, CNRS, UMR 7598, Laboratoire Jacques-Louis LionsParisFrance
  2. 2.Institut für MathematikTechnische Universität BerlinBerlinGermany

Personalised recommendations