Nonlinear Frames and Sparse Reconstructions in Banach Spaces

Abstract

In the first part of this paper, we consider nonlinear extension of frame theory by introducing bi-Lipschitz maps F between Banach spaces. Our linear model of bi-Lipschitz maps is the analysis operator associated with Hilbert frames, p-frames, Banach frames, g-frames and fusion frames. In general Banach space setting, stable algorithms to reconstruct a signal x from its noisy measurement $F(x)+\epsilon$ may not exist. In this paper, we establish exponential convergence of two iterative reconstruction algorithms when F is not too far from some bounded below linear operator with bounded pseudo-inverse, and when F is a well-localized map between two Banach spaces with dense Hilbert subspaces. The crucial step to prove the latter conclusion is a novel fixed point theorem for a well-localized map on a Banach space. In the second part of this paper, we consider stable reconstruction of sparse signals in a union $\mathbf{A}$ of closed linear subspaces of a Hilbert space $\mathbf{H}$ from their nonlinear measurements. We introduce an optimization framework called a sparse approximation triple $(\mathbf{A}, \mathbf{M}, \mathbf{H})$, and show that the minimizer

$$\begin{aligned} x^*=\mathrm{argmin}_{\hat{x}\in {\mathbf M}\ \mathrm{with} \ \Vert F(\hat{x})-F(x^0)\Vert \le \epsilon } \Vert \hat{x}\Vert _{\mathbf M} \end{aligned}$$

provides a suboptimal approximation to the original sparse signal $x^0\in \mathbf{A}$ when the measurement map F has the sparse Riesz property and the almost linear property on ${\mathbf A}$. The above two new properties are shown to be satisfied when F is not far away from a linear measurement operator T having the restricted isometry property.

Appendices

Appendix 1: Bi-Lipschitz Map and Uniform Stability

In this appendix, we provide some sufficient conditions, mostly optimal, for a differentiable map to have the bi-Lipschitz property (1.1), see Theorems 6.3 and 6.5 in Banach space setting, and Theorems 6.7 and 6.9 in Hilbert space setting.

For a differentiable map F from one Banach space ${\mathbf B}_1$ to another Banach space ${\mathbf B}_2$ that has the bi-Lipschitz property (1.1), we have

$$\begin{aligned} A \Vert y\Vert \le \frac{\Vert F(x+ty)-F(x)\Vert }{t} \le B \Vert y\Vert \quad \mathrm{for \ all} \ \ x, y\in {\mathbf B}_1\ \mathrm{and} \ t>0, \end{aligned}$$

where A, B are the constants in the bi-Lipschitz property (1.1). Then taking limit as $t\rightarrow 0$ leads to a necessary condition for a differentiable bi-Lipschitz map.

Theorem 6.1

Let ${\mathbf B}_1$ and ${\mathbf B}_2$ be Banach spaces. If $F:{\mathbf B}_1\rightarrow {\mathbf B}_2$ is a differentiable map that has the bi-Lipschitz property (1.1), then its derivative $F'(x), x\in {\mathbf B}_1$, has the uniform stability property (1.2).

For $\mathbf{B}_1=\mathbf{B}_2={\mathbb R}$, a differentiable map F with the uniform stability property (1.2) for its derivative has the bi-Lipschitz property (1.1), but it is not true in general Banach space setting. Maps $E_{p, \epsilon }, 1\le p\le \infty , \epsilon \in [0, \pi /4)$, from ${\mathbb R}$ to ${\mathbb R}^2$ in the example below are such examples.

Example 6.2

For $1\le p\le \infty$ and $\epsilon \in [0, \pi /4)$, define $E_{p, \epsilon }: {\mathbb R}\longmapsto {\mathbb R}^2$ by

$$\begin{aligned} E_{p, \epsilon }(t)=\left\{ \begin{array}{l} (-\cos \epsilon , \sin \epsilon )- (\sin \epsilon , \cos \epsilon )(t+\pi /2+\epsilon )\\ \mathrm{if} \ t\in (-\infty , -\pi /2-\epsilon ),\\ (\sin t, -\cos t) \mathrm{if} \ t\in [-\pi /2-\epsilon , \pi /2+\epsilon ],\\ (\cos \epsilon , \sin \epsilon )+ (-\sin \epsilon , \cos \epsilon )(t-\pi /2-\epsilon )\\ \mathrm{if} \ t\in (\pi /2+\epsilon , \infty ), \end{array}\right. \end{aligned}$$

(6.1)

see Fig. 1. The maps $E_{p, \epsilon }$ just defined do not have the bi-Lipschitz property (1.1), but their derivatives $E_{p, \epsilon }^\prime$ have the uniform stability property (1.2),

$$\begin{aligned} \frac{\sqrt{2}}{2} |\tilde{t}|\le & {} \Vert E_{p, \epsilon }^\prime (t) \tilde{t}\Vert _p=\left\{ \begin{array}{ll} \Vert (\tilde{t}\sin \epsilon , \tilde{t}\cos \epsilon )\Vert _p &{} \mathrm{if} \ t< -\pi /2-\epsilon \\ \Vert (\tilde{t}\cos t, \tilde{t}\sin t)\Vert _p &{} \mathrm{if} \ |t|\le \pi /2+\epsilon \\ \Vert (-\tilde{t} \sin \epsilon , \tilde{t} \cos \epsilon )\Vert _p &{} \mathrm{if} \ t>\pi /2+\epsilon \end{array}\right. \nonumber \\\le & {} 2|\tilde{t}|\ \ \mathrm{for \ all} \ t,\tilde{t}\in {\mathbb R}, \end{aligned}$$

where $\Vert \cdot \Vert _p, 1\le p\le \infty$, is the p-norm on the Euclidean space ${\mathbb R}^2$.

Fig. 1

Maps $E_{p, \epsilon }$ from ${\mathbb R}$ to ${\mathbb R}^2$ with $\epsilon =0$ (left) and $\epsilon =\pi /6$ (right)

Given a differentiable bi-Lipschitz map F from one Banach space ${\mathbf B}_1$ to another Banach space ${\mathbf B}_2$ such that its derivative $F'(x)$ is uniformly stable, define

$$\begin{aligned} \alpha _F:=\sup _{\Vert y\Vert =1}\inf _{\Vert z\Vert =1}\sup _{x\in {\mathbf B}_1}\Big \Vert \frac{F'(x)y}{\Vert F'(x)y\Vert }-z\Big \Vert . \end{aligned}$$

(6.2)

The quantity $\alpha _F$ is the minimal radius such that for any $0\ne y\in {\mathbf B}_1$, the set ${\mathbb B}(y)$ of unit vectors $F'(x)y/\Vert F'(x)y\Vert , x\in {\mathbf B}_1$, is contained in a ball of radius $\alpha _F<1$ centered at a unit vector. Our next theorem shows that a differentiable bi-Lipschitz map F with its derivative $F'(x)$ being uniformly stable and continuous and with $\alpha _F$ in (6.2) satisfying $\alpha _F<1$ has the bi-Lipschitz property (1.1).

Theorem 6.3

Let ${\mathbf B}_1$ and ${\mathbf B}_2$ be Banach spaces, and F be a continuously differentiable map from ${\mathbf B}_1$ to ${\mathbf B}_2$ with the property that its derivative has the uniform stability property (1.2). If $\alpha _F$ in (6.2) satisfies

$$\begin{aligned} \alpha _F<1,\end{aligned}$$

(6.3)

then F is a bi-Lipschitz map.

Proof

Given $x, y\in {\mathbf B}_1$ with $y\ne 0$,

$$\begin{aligned} F(x+y)-F(x)= & {} \int _0^1 F'(x+ty) y dt = \left( \int _0^1 \Vert F'(x+ty)y\Vert dt\right) z\\&+ \int _0^1 \Vert F'(x+ty)y\Vert \left( \frac{F'(x+ty)y}{\Vert F'(x+ty)y\Vert }-z \right) dt, \end{aligned}$$

where $z\in {\mathbf B}_2$ with $\Vert z\Vert =1$. Thus

$$\begin{aligned} \Vert F(x+y)-F(x)\Vert\ge & {} \left( \int _0^1 \Vert F'(x+ty)y\Vert dt\right) \\&\quad \times \left( 1-\inf _{\Vert z\Vert =1} \sup _{0\le t\le 1}\left\| \frac{F'(x+ty)y}{\Vert F'(x+ty)y\Vert }-z \right\| \right) \\\ge & {} (1-\alpha _F) \left( \int _0^1 \Vert F'(x+ty)y\Vert dt\right) \ge (1-\alpha _F) A\Vert y\Vert , \end{aligned}$$

and

$$\begin{aligned} \Vert F(x+y)-F(x)\Vert \le \int _0^1 \Vert F'(x+ty)y\Vert dt\le B\Vert y\Vert , \end{aligned}$$

where A, B are lower and upper stability bounds in the uniform stability property (1.2). Combining the above two estimates completes the proof. $\square$

Remark 6.4

The U-shaped map $E_{p, \epsilon }$ in Example 6.2 with $p=\infty$ and $\epsilon =0$ is not a bi-Lipschitz map and

$$\begin{aligned} \alpha _{E_{\infty , 0}}= & {} \inf _{\Vert z\Vert _\infty =1} \sup _{|t|\le \pi /2}\left\| \frac{(\cos t, \sin t)}{\max (|\cos t|, |\sin t|)}-z\right\| _\infty \\= & {} \sup _{|t|\le \pi /2}\left\| \frac{(\cos t, \sin t)}{\max (|\cos t|, |\sin t|)}-(1, 0)\right\| _\infty =1. \end{aligned}$$

This indicates that the geometric condition (6.3) about $\alpha _F$ is optimal.

For a differentiable map F not far away from a bounded below linear operator T, we suggest using $Ty/\Vert Ty\Vert$ as the center of the ball containing the set of unit vectors $F'(x) y/\Vert F'(x) y\Vert , x\in \mathbf{B}_1$, and define the minimal radius of that ball by $\beta _{F,T}$ in (1.5). Then obviously

$$\begin{aligned} \alpha _F\le \beta _{F,T}. \end{aligned}$$

(6.4)

This together with Theorem 6.3 implies that a differentiable map F satisfying $\beta _{F, T}<1$ is a bi-Lipschitz map.

Theorem 6.5

Let ${\mathbf B}_1$ and ${\mathbf B}_2$ be Banach spaces, and F be a continuously differentiable map from ${\mathbf B}_1$ to ${\mathbf B}_2$ with its derivative having the uniform stability property (1.2). If $T\in {\mathcal B}({\mathbf B}_1, {\mathbf B}_2)$ is bounded below and satisfies (1.4), then F is a bi-Lipschitz map.

We may use the following quantity to measure the distance between differentiable map F and bounded below linear operator T,

$$\begin{aligned} \delta _{F, T}:=\sup _{0\ne y\in {\mathbf B}_1} \sup _{x\in {\mathbf B}_1}\frac{\Vert F(x+y)-F(x)-Ty\Vert }{\Vert Ty\Vert }= \sup _{0\ne y\in {\mathbf B}_1} \sup _{z\in {\mathbf B}_1}\frac{\Vert F'(z)y-Ty\Vert }{\Vert Ty\Vert }. \end{aligned}$$

(6.5)

By direct computation,

$$\begin{aligned} \beta _{F, T} \le \sup _{\Vert y\Vert =1} \sup _{x\in {\mathbf B}_1} \Big (\frac{\Vert F'(x)y-Ty\Vert }{\Vert F'(x)y\Vert }+\frac{\big |\Vert F'(x)y\Vert -\Vert Ty\Vert \big |}{\Vert F'(x)y\Vert }\Big )\le \frac{2\delta _{F,T}}{1-\delta _{F,T}}. \end{aligned}$$

Thus the geometric condition (1.4) in Theorem 6.5 can be replaced by the condition $\delta _{F,T}<1/3$.

Corollary 6.6

Let ${\mathbf B}_1, {\mathbf B}_2, F$ and T be as in Theorem 6.5. If $\delta _{F,T}<1/3$, then F is a bi-Lipschitz map.

The geometric condition (1.4) to guarantee the bi-Lipschitz property for the map F is optimal in general Banach space setting, as $\beta _{E_{\infty , 0}, T_1}=1$ for the U-shaped map $E_{\infty , 0}$ in Example 6.2 and the linear operator $T_1t:=(t,0), t\in {\mathbb R}$. But in Hilbert space setting, as shown in the next theorem, the geometric condition (1.4) could be relaxed to $\beta _{F,T}<\sqrt{2}$.

Theorem 6.7

Let ${\mathbf H}_1$ and ${\mathbf H}_2$ be Hilbert spaces, and let $F:{\mathbf H}_1\rightarrow {\mathbf H}_2$ be a continuously differentiable map with its derivative having the uniform stability property (1.2). If there exists a linear operator $T\in {\mathcal B}({\mathbf H}_1, {\mathbf H}_2)$ satisfying (1.3) and (1.6), then F is a bi-Lipschitz map.

Proof

Take $u, v\in {\mathbf H}_1$ with $v\ne 0$. Then

$$\begin{aligned} \Vert F(u+v)-F(u)\Vert \le \int _0^1 \Vert F'(u+tv) v\Vert dt\le B\Vert v\Vert , \end{aligned}$$

(6.6)

where B is the upper stability bound in (1.2). Observe that

$$\begin{aligned} \langle F'(u)v, Tv\rangle = \Vert F'(u) v\Vert \Vert Tv\Vert \left( 1-\frac{1}{2} \Big \Vert \frac{F'(u)v}{\Vert F'(u)v\Vert }-\frac{Tv}{\Vert Tv\Vert }\Big \Vert ^2\right) . \end{aligned}$$

Then

$$\begin{aligned} \langle F'(u)v, Tv\rangle \ge \frac{2-(\beta _{F,T})^2}{2} \Vert F'(u) v\Vert \Vert Tv\Vert , \end{aligned}$$

which implies that

$$\begin{aligned} \langle F(u+v)-F(u), Tv\rangle= & {} \int _0^1 \langle F'(u+tv)v, Tv\rangle dt\nonumber \\\ge & {} \frac{2-(\beta _{F,T})^2}{2} \left( \int _0^1 \Vert F'(u+tv) v\Vert dt\right) \Vert Tv\Vert \nonumber \\\ge & {} \frac{2-(\beta _{F,T})^2}{2}A \Vert Tv\Vert \Vert v\Vert , \end{aligned}$$

(6.7)

where A is the lower stability bound in (1.2). Hence

$$\begin{aligned} \Vert F(u+v)-F(u)\Vert \ge \frac{\langle F(u+v)-F(u), Tv\rangle }{\Vert Tv\Vert } \ge \frac{2-( \beta _{F,T})^2}{2}A \Vert v\Vert . \end{aligned}$$

(6.8)

Combining (6.6) and (6.8) proves the bi-Lipschitz property for F. $\square$

Remark 6.8

The geometric condition (1.6) is optimal as for the U-shaped map $E_{p, \epsilon }$ in Example 6.2 with $p=2$ and $\epsilon =0$,

$$\begin{aligned} \beta _{E_{2, 0}, T_1}= \sup _{\tilde{t}\ne 0} \sup _{|t|\le \pi /2}\left\| \frac{(\tilde{t}\cos t, \tilde{t}\sin t)}{(\cos ^2 t+\sin ^2 t)^{1/2} |\tilde{t}|}-\frac{(\tilde{t},0)}{|\tilde{t}|}\right\| _2=\sqrt{2} \end{aligned}$$

(6.9)

where $T_1\tilde{t}=(\tilde{t}, 0), \tilde{t}\in {\mathbb R}$.

Define

$$\begin{aligned} \theta _{F,T}=\sup _{u\in {\mathbf H}_1, v\ne 0} \arccos \left( \frac{\langle F'(u) v, Tv\rangle }{\Vert F'(u)v\Vert \Vert Tv\Vert }\right) , \end{aligned}$$

the maximal angle between vectors $F'(u)v$ and Tv in the Hilbert space ${\mathbf H}_2$. Then

$$\begin{aligned} \beta _{F, T} =2\sin \frac{\theta _{F,T}}{2}. \end{aligned}$$

So the geometric condition (1.6) can be interpreted as that the angles between $F'(u)v$ and Tv are less than or equal to $\theta _{F,T}\in [0, \pi /2)$ for all $u, v\in {\mathbf H}_1$. The above equivalence between the geometric condition (1.6) and the angle condition $\theta _{F, T}<\pi /2$, together with (1.2) and (1.3), implies the existence of positive constants $A_1, B_1$ such that

$$\begin{aligned} A_1 \Vert Tv\Vert ^2 \le \langle F'(u)v, Tv\rangle \le B_1 \Vert Tv\Vert ^2, \ u,v\in {\mathbf H}_1. \end{aligned}$$

(6.10)

The converse can be proved to be true too. Thus $\beta _{F, T}<\sqrt{2}$ if and only if $S:=T^*F$ is strictly monotonic. Here a bounded map S on a Hilbert space ${\mathbf H}$ is said to be strictly monotonic [53] if there exist positive constants m and M such that

$$\begin{aligned} m \Vert u-v\Vert ^2\le \langle u-v, S(u)-S(v)\rangle \le M \Vert u-v\Vert ^2 \ \mathrm{for \ all} \ u,v\in {\mathbf H}. \end{aligned}$$

As an application of the above equivalence, Theorem 6.7 can be reformulated as follows.

Theorem 6.9

Let ${\mathbf H}_1$ and ${\mathbf H}_2$ be Hilbert spaces, and let $F:{\mathbf H}_1\rightarrow {\mathbf H}_2$ be a continuously differentiable map with its derivative having the uniform stability property (1.2). If there exists a linear operator $T\in {\mathcal B}({\mathbf H}_1, {\mathbf H}_2)$ satisfying (1.3) and (6.10), then F is a bi-Lipschitz map.

From Theorem 6.9 we obtain the following result similar to the one in Corollary 6.6.

Corollary 6.10

Let ${\mathbf H}_1, {\mathbf H}_2$ and F be as in Theorem 6.9. If there exists a bounded below linear operator $T\in {\mathcal B}({\mathbf H}_1, {\mathbf H}_2)$ with $\delta _{F,T}<\sqrt{2}-1$, then F is a bi-Lipschitz map.

Given a differentiable map F, it is quite technical in general to construct linear operator T satisfying (1.3) and (1.4) in Banach space setting (respectively (1.3) and (6.10) in Hilbert space setting). A conventional selection is that $T=F'(x_0)$ for some $x_0\in {\mathbf B}_1$, but such a selection is not always favorable. Let $\Phi =(\phi _\lambda )_{\lambda \in \Lambda }$ be impulse response vector with its entry $\phi _\lambda$ being the impulse response of the signal generating device at the innovation position $\lambda \in \Lambda$, and $\Psi =(\psi _\gamma )_{\gamma \in \Gamma }$ be sampling functional vector with entry $\psi _\gamma$ reflecting the characteristics of the acquisition device at the sampling position $\gamma \in \Gamma$. In order to consider bi-Lipschitz property of the nonlinear sampling map

$$\begin{aligned} S_{f, \Phi , \Psi }: \ell ^2(\Lambda )\ni x \longmapsto x^T\Phi \overset{\mathrm{companding }}{\longmapsto }f(x^T\Phi ) \overset{\mathrm{sampling}}{\longmapsto }\langle f(x^T\Phi ), \Psi \rangle \in \ell ^2(\Gamma ) \end{aligned}$$

related to instantaneous companding $h(t)\longmapsto f(h(t))$, a linear operator

$$\begin{aligned} T:= A_{\Phi , \Phi } \left( A_{\Phi , \Psi } (A_{\Psi , \Psi })^{-1} A_{\Psi , \Phi }\right) ^{-1} A_{\Phi , \Psi } (A_{\Psi , \Psi })^{-1} \end{aligned}$$

satisfying (1.3) and (6.10) is implicitly introduced in [49], where

$$\begin{aligned} A_{\Phi , \Psi }=(\langle \phi _\lambda , \psi _\gamma \rangle )_{\lambda \in \Lambda , \gamma \in \Gamma } \end{aligned}$$

is the inter-correction matrix between $\Phi$ and $\Psi$.

Appendix 2: Sparse Approximation Triple

In this appendix, we establish convergence of the greedy algorithm (5.3) in the Banach space ${\mathbf M}$, and use it to estimate an approximation error in the Hilbert space ${\mathbf H}_1$.

1.1 (a) Greedy Algorithm

In this subsection, we show that the greedy algorithm (5.3) converges, which play an important role in the proofs of Theorems 5.1 and 7.2.

Theorem 7.1

Let $({\mathbf A}, {\mathbf M}, {\mathbf H}_1)$ be a sparse approximation triple. Then $x_{\mathbf A, \mathbf M}^k, k\ge 0$, in the greedy algorithm (5.3) converges to $x\in {\mathbf M}$,

$$\begin{aligned} \lim _{k\rightarrow \infty } \left\| x_{\mathbf A, \mathbf M}^k-x \right\| _{\mathbf M}=0. \end{aligned}$$

(7.1)

Proof

The convergence of $x_{\mathbf A, \mathbf M}^k, k\ge 0$, follows from

$$\begin{aligned} \sum _{k=0}^K\left\| x_{\mathbf A, \mathbf M}^{k+1}-x_{\mathbf A, \mathbf M}^k\right\| _{\mathbf M}= \Vert x\Vert _{\mathbf M}-\left\| x-x_{\mathbf A, \mathbf M}^{K+1}\right\| _{\mathbf M}\le \Vert x\Vert _{\mathbf M}, \ K\ge 0, \end{aligned}$$

(7.2)

which is obtained by applying the norm splitting property (1.16) recursively.

Denote by $x_{\mathbf A, \mathbf M}^\infty \in {\mathbf M}$ the limit of $x_{\mathbf A, \mathbf M}^{k}, k\ge 0$. Set

$$\begin{aligned} z=\mathrm{argmin}_{\hat{x}\in {\mathbf A}} \left\| x-x_{\mathbf A, \mathbf M}^\infty -\hat{x}\right\| _{\mathbf M}. \end{aligned}$$

Then for all $k\ge 1$,

$$\begin{aligned} \left\| x-x_{\mathbf A, \mathbf M}^{k+1}\right\| _{\mathbf M}\le & {} \left\| x-x_{\mathbf A, \mathbf M}^{k}-z\right\| _{\mathbf M} \le \left\| x-x_{\mathbf A, \mathbf M}^{\infty }-z\right\| _{\mathbf M} + \left\| x_{\mathbf A, \mathbf M}^{\infty }- x_{\mathbf A, \mathbf M}^{k}\right\| _{\mathbf M} \\= & {} \left\| x-x_{\mathbf A, \mathbf M}^{\infty }\right\| _{\mathbf M}- \Vert z\Vert _{\mathbf M}+ \left\| x_{\mathbf A, \mathbf M}^{\infty }- x_{\mathbf A, \mathbf M}^{k}\right\| _{\mathbf M}, \end{aligned}$$

where the first inequality follows from (5.3) and the last equality holds by the norm splitting property (1.16). Therefore

$$\begin{aligned} \Vert z\Vert \le \left\| x_{\mathbf A, \mathbf M}^{\infty }- x_{\mathbf A, \mathbf M}^{k+1}\right\| _{\mathbf M} + \left\| x_{\mathbf A, \mathbf M}^{\infty }- x_{\mathbf A, \mathbf M}^{k}\right\| _{\mathbf M} \rightarrow 0 \ \ \mathrm{as} \ \ k\rightarrow \infty , \end{aligned}$$

which implies that

$$\begin{aligned} 0= \mathrm{argmin}_{\hat{x}\in {\mathbf A}} \left\| x-x_{\mathbf A, \mathbf M}^\infty -\hat{x} \right\| _{\mathbf M}. \end{aligned}$$

Hence the projection of $x-x_{\mathbf A, \mathbf M}^\infty$ onto ${\mathbf A}_i$ are zero for all $i\in I$ by the common best approximator property (1.15). This proves the following consistency condition,

$$\begin{aligned} \langle x_{\mathbf A, \mathbf M}^\infty , y\rangle =\langle x, y\rangle \ \mathrm{for \ all} \ y\in {\mathbf A}. \end{aligned}$$

(7.3)

From the consistency property (7.3), we conclude that (7.3) hold for all $y\in k{\mathbf A}, k\ge 0$, and hence for all y in the closure of $\cup _{k\ge 0} k{\mathbf A}$. This together with the sparse density property (v) of the sparse approximation triple $(\mathbf{A}, \mathbf{M}, \mathbf{H}_1)$ proves that $x_{\mathbf A, \mathbf M}^\infty = x$, and hence the convergence (7.1) of $x_{\mathbf A, \mathbf M}^k, k\ge 0$, to $x\in {\mathbf M}$. $\square$

Given a sparse approximation triple $(\mathbf{A}, \mathbf{M}, \mathbf{H}_1)$, we say that $x\in {\mathbf M}$ is compressible [7, 24, 26, 40] if $\{\sigma _{k{\mathbf A}, \mathbf{M}}(x)\}_{k=1}^\infty$ having rapid decay, such as

$$\begin{aligned} \sigma _{k{\mathbf A}, \mathbf{M}}(x)\le C k^{-\alpha }\ \mathrm{for \ some} \ C,\ \alpha >0, \end{aligned}$$

where $\sigma _{k{\mathbf A}, {\mathbf M}}(x)$ is the best approximation error of x from $k{\mathbf A}$,

$$\begin{aligned} \sigma _{k{\mathbf A}, {\mathbf M}}(x) :=\inf _{\hat{x}\in k{\mathbf A}}\Vert \hat{x}-x\Vert _{\mathbf M}, \ k\ge 1. \end{aligned}$$

(7.4)

For the sequence $x_{\mathbf A, \mathbf M}^k, k\ge 0$, in the greedy algorithm (5.3), we have

$$\begin{aligned} \left\| x_{\mathbf A, \mathbf M}^k-x \right\| _{\mathbf M}\ge \sigma _{k{\mathbf A}, \mathbf M}(x), \end{aligned}$$

(7.5)

as $x_{\mathbf A, \mathbf M}^k\in k{\mathbf A}, k\ge 1.$ The above inequality becomes an equality in the classical sparse recovery setting. We do not know whether and when the greedy algorithm (5.3) is suboptimal, i.e., there exists a positive constant C such that

$$\begin{aligned} \left\| x_{\mathbf A, \mathbf M}^k-x \right\| _{\mathbf M} \le C \sigma _{k{\mathbf A}, \mathbf M}(x),\ x\in M, \end{aligned}$$

(7.6)

even for compressible signals. The reader may refer to [52] for the study of various greedy algorithms.

1.2 (b) Sparse Approximation

In this subsection, we apply Theorem 7.1 to estimate an approximation error, which has been used in the proof of Theorem 4.1.

Theorem 7.2

Let $(\mathbf{A}, \mathbf{M}, \mathbf{H}_1)$ be a sparse approximation triple and $a_\mathbf{A}$ be as in (1.19). Then

$$\begin{aligned} \Vert x-x_{\mathbf A, \mathbf M}\Vert _{{\mathbf H}_1} \le \sqrt{a_{\mathbf A}} \Vert x\Vert _{\mathbf M}, \ x\in {\mathbf M}, \end{aligned}$$

(7.7)

where $x_{\mathbf A, \mathbf M}$ is a best approximator of $x\in \mathbf{M}$.

Proof

Take $0\ne x\in {\mathbf M}$ and let $x_{\mathbf A, \mathbf M}^k, k\ge 0$, be as in the greedy algorithm (5.3). Write $u_k=x_{\mathbf A, \mathbf M}^{k+1}-x_{\mathbf A, \mathbf M}^{k}, k\ge 0$. Thus

$$\begin{aligned} u_k=\mathrm{argmin}_{\hat{x}\in {\mathbf A}} \Vert x-x_{{\mathbf A},{\mathbf M}}^k-\hat{x}\Vert _{\mathbf M}= \mathrm{argmin}_{\hat{x}\in {\mathbf A}} \Vert (x-x_{{\mathbf A},{\mathbf M}}^{k-1})-u_{k-1}-\hat{x}\Vert _{\mathbf M} \in {\mathbf A}, \end{aligned}$$

and

$$\begin{aligned} x-x_{\mathbf A, \mathbf M}=\sum _{k\ge 1} u_k \end{aligned}$$

by Theorem 7.1. This together with (1.16), (1.19), (7.1) and (7.2) implies that

$$\begin{aligned} \Vert x-x_{\mathbf A, \mathbf M}\Vert _{{\mathbf H}_1}\le \sum _{k\ge 1} \Vert u_k\Vert _{{\mathbf H}_1}\le \sqrt{a_{\mathbf A}} \sum _{k\ge 1} \Vert u_{k-1}\Vert _{\mathbf M}\le \sqrt{a_{\mathbf A}}\Vert x\Vert _{\mathbf M}. \end{aligned}$$

This completes the proof. $\square$