The finite steps of convergence of the fast thresholding algorithms with f-feedbacks in compressed sensing

Abstract

Iterative algorithms based on thresholding, feedback, and null space tuning (NST+HT+FB) for sparse signal recovery are exceedingly effective and efficient, particularly for large-scale problems. The core algorithm is shown to converge in finitely many steps under a (preconditioned) restricted isometry condition. We derive in this article the number of iterations to guarantee the convergence of the NST+HT+FB algorithm. Moreover, an accelerated class of adaptive feedback scheme of the iterative algorithm, termed NST+HT+f -FB, is proposed and analyzed. The scheme NST+HT+f -FB has a variable/adaptive index selection and different feedback principles at each iteration defined by a function f(k). It is even more effective, both from its ability to recover sparse signals with a larger number of non-zeros and from its rate of convergence. The convergence of the accelerated scheme is established. The finite number of iterations for guaranteed convergence by the NST+HT+f -FB scheme is also obtained. Furthermore, it is possible to accelerate the rate of convergence and improve the condition of convergence by selecting an appropriate size of the thresholding index set T_k per iteration. The theoretical findings are validated through extensive numerical experiments. It has been shown that the proposed algorithm has a clearly advantageous balance of efficiency, adaptivity, and accuracy compared with other state-of-the-art greedy algorithms. Detailed comparisons are provided.

Appendix:

1.1 Preliminary results

Definition 7.1

[2]. For each integer s = 1, 2,..., the matrix A is said to satisfy the RIP of order s with constant δ_s if

$$ \left( 1-\delta_{s}\right)\|x\|_{2}^{2}\leq\|Ax\|_{2}^{2}\leq\left( 1+\delta_{s}\right)\|x\|_{2}^{2} $$

holds for all s-sparse vectors x. Equivalently, it is given by

$$ \delta_{s}=\max_{|S|\leq s}\|I-A_{S}^{\ast}A_{S}\|_{2}. $$

Definition 7.2

[50]. For each integer s = 1, 2,..., the matrix A is said to satisfy the P-RIP of order s with constant γ_s if

$$ (1-\gamma_{s})\|x\|_{2}^{2}\leq\|(AA^{\ast})^{-\frac{1}{2}}Ax\|_{2}^{2} $$

holds for all s-sparse vectors x. In fact, the preconditioned restricted isometry constant γ_s characterizes the restricted isometry property of the preconditioned matrix $(AA^{\ast })^{-\frac {1}{2}}A$. Since

$$ \|(AA^{\ast})^{-\frac{1}{2}}Ax\|_{2}\leq\|(AA^{\ast})^{-\frac{1}{2}}A\|_{2}\|x\|_{2}=\|x\|_{2}, $$

γ_s is actually the smallest number such that, for all s-sparse vectors x,

$$ (1-\gamma_{s})\|x\|_{2}^{2}\leq\|(AA^{\ast})^{-\frac{1}{2}}Ax\|_{2}^{2}\leq(1+\gamma_{s})\|x\|_{2}^{2}. $$

It indicates $\gamma _{s}(A)=\delta _{s}\left (\left (AA^{\ast }\right )^{-\frac {1}{2}}A\right )$. Equivalently, it is given by

$$ \gamma_{s}=\max_{|S|\leq s}\|I-A_{S}^{\ast}\left( AA^{\ast}\right)^{-1}A_{S}\|_{2}. $$

Lemma 7.2

Let δ_t be the RIP constant of A. For $u,v\in \mathbb {C}^{N}$, if |supp(u) ∪ supp(v)|≤ t, then $|\langle u,\left (I-A^{\ast }A\right )v\rangle |\leq \delta _{t}\|u\|_{2}\|v\|_{2}$. Suppose $|T^{\prime } \cup supp (v)|\leq t$, then $\|\left (\left (I-A^{\ast }A\right )v\right )_{T^{\prime }}\|_{2}\leq \delta _{t}\|v\|_{2}$.

Proof

Setting T = supp(v) ∪ supp(u), |T|≤ t, one has

$$ \begin{aligned} |\langle u,\left( I-A^{\ast}A\right)v\rangle| & =|\langle u,v\rangle-\langle Au,Av\rangle| \\ &=|\langle u_{T},v_{T}\rangle-\langle A_{T}u_{T},A_{T}v_{T}\rangle|\\ &=|\langle u_{T},(I-A_{T}^{\ast}A_{T})v_{T}\rangle|\\ &\leq\|u_{T}\|_{2}\|(I-A_{T}^{\ast}A_{T})v_{T}\|_{2}\\ &\leq\|u_{T}\|_{2}\|I-A_{T}^{\ast}A_{T}\|_{2}\|v_{T}\|_{2}\\ &\leq\delta_{t}\|u\|_{2}\|v\|_{2}. \end{aligned} $$

The first inequality is due to the Cauchy-Schwarz inequality and the second inequality is due to the submultiplicativity of matrix norms, while the last step is based on Definition 7.1. Since

$$ \begin{aligned} \|\left( \left( I-A^{\ast}A\right)v\right)_{T^{\prime}}\|_{2}^{2} &=\langle\left( \left( I-A^{\ast}A\right)v\right)_{T^{\prime}},\left( I-A^{\ast}A\right)v\rangle\\ &\leq\delta_{t}\|\left( \left( I-A^{\ast}A\right)v\right)_{T^{\prime}}\|_{2}\|v\|_{2}, \end{aligned} $$

one can obtain that $\|\left (\left (I-A^{\ast }A\right )v\right )_{T^{\prime }}\|_{2}\leq \delta _{t}\|v\|_{2}$. □

Remark 7.1

Let γ_t be the P-RIP constant of A, i.e., $\gamma _{t}(A)=\delta _{t}\left (\left (AA^{\ast }\right )^{-\frac {1}{2}}A\right )$. For $u,v\in \mathbb {C}^{N}$, if |supp(u) ∪ supp(v)|≤ t, then $|\langle u,\left (I-A^{\ast }\left (AA^{\ast }\right )^{-1}A\right )v\rangle |\leq \gamma _{t}\|u\|_{2}\|v\|_{2}$. Suppose $|T^{\prime } \cup supp (v)|\leq t$, then $\|\left (\left (I-A^{\ast }\left (AA^{\ast }\right )^{-1}A\right )v\right )_{T^{\prime }}\|_{2}\leq \gamma _{t}\|v\|_{2}$.

Lemma 7.2

For $e\in \mathbb {C}^{M}$, $\|\left (A^{\ast }e\right )_{T}\|_{2}\leq \sqrt {1+\delta _{t}}\|e\|_{2}$, when |T|≤ t.

Proof

$$ \begin{aligned} \|\left( A^{\ast}e\right)_{T}\|_{2}^{2}&=\langle A^{\ast}e,\left( A^{\ast}e\right)_{T}\rangle=\langle e,A\left( A^{\ast}e\right)_{T}\rangle\\ &\leq\|e\|_{2}\|A\left( A^{\ast}e\right)_{T}\|_{2}\leq\|e\|_{2}\sqrt{1+\delta_{t}}\|\left( A^{\ast}e\right)_{T}\|_{2}, \end{aligned} $$

Hence, for all $e\in \mathbb {C}^{M}$, we have $\|\left (A^{\ast }e\right )_{T}\|_{2}\leq \sqrt {1+\delta _{t}}\|e\|_{2}$. □

Remark 7.2

If $\delta _{t}\left (\left (AA^{\ast }\right )^{-1}A\right )=\theta _{t}$, then for $e\in \mathbb {C}^{M}$, $\|\left (A^{\ast }\left (AA^{\ast }\right )^{-1}e\right )_{T}\|_{2}\leq \sqrt {1+\theta _{t}}\|e\|_{2}$, when |T|≤ t.

1.2 Proof of Lemma 3.1.

Proof

We first have that

$\|[\mu ^{\prime }+A^{\ast }(AA^{\ast })^{-1}(y-A\mu ^{\prime })]_{T}\|_{2}\geq \|[\mu ^{\prime }+A^{\ast }(AA^{\ast })^{-1}(y-A\mu ^{\prime })]_{S}\|_{2}.$

Eliminating the common terms over $T\bigcap S$, one has

$\|[\mu ^{\prime }+A^{\ast }(AA^{\ast })^{-1}(y-A\mu ^{\prime })]_{T\setminus S}\|_{2}\geq \|[\mu ^{\prime }+A^{\ast }(AA^{\ast })^{-1}(y-A\mu ^{\prime })]_{S\setminus T}\|_{2}.$

For the left hand,

$$ \begin{aligned} &\|[\mu^{\prime}+A^{\ast}(AA^{\ast})^{-1}(y-A\mu^{\prime})]_{T\setminus S}\|_{2}\\ =&\|[\mu^{\prime}-x+A^{\ast}(AA^{\ast})^{-1}(Ax+e-A\mu^{\prime})]_{T\setminus S}\|_{2}\\ =&\|[(I-A^{\ast}(AA^{\ast})^{-1}A)(\mu^{\prime}-x)+A^{\ast}(AA^{\ast})^{-1}e]_{T\setminus S}\|_{2}. \end{aligned} $$

The right hand satisfies

$$ \begin{aligned} &\|[\mu^{\prime}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( y-A\mu^{\prime}\right)]_{S\setminus T}\|_{2}\\ =&\|[\mu^{\prime}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( Ax+e-A\mu^{\prime}\right)+x-x]_{S\setminus T}\|_{2}\\ \geq&\|x_{S\setminus T}\|_{2}-\|[\left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\prime}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e]_{S\setminus T}\|_{2}. \end{aligned} $$

Consequently,

$$ \begin{aligned} &\|x_{S\setminus T}\|_{2}\\ \leq& \|[\left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\prime}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e]_{S\setminus T}\|_{2}\\ &+\|[\left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\prime}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e]_{T\setminus S}\|_{2}\\ \leq&\sqrt{2}\|[\left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\prime}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e]_{T\triangle S}\|_{2}\\ \leq&\sqrt{2}\|[\left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\prime}-x\right)]_{T\triangle S}\|_{2}+\sqrt{2}\|[A^{\ast}\left( AA^{\ast}\right)^{-1}e]_{T\triangle S}\|_{2}\\ \leq&\sqrt{2}\left( \gamma_{s+s^{\prime}+t}\|x-\mu^{\prime}\|_{2}+\sqrt{1+\theta_{t+s}}\|e\|_{2}\right). \end{aligned} $$

The last step is due to Remark 7.1 and Remark 7.2. □

1.3 Proof of Lemma 3.2.

Proof

For any $z\in \mathbb {C}^{N}$ supported on T,

$$ \begin{aligned} \langle A\mu^{\prime}-y,Az\rangle&=\langle A_{T}x_{T}^{\prime}+A_{T}(A_{T}^{\ast} A_{T})^{-1}A_{T}^{\ast} A_{T^{c}}x_{T^{c}}^{\prime}-y,A_{T}z_{T}\rangle\\ &=\langle A_{T}^{\ast}\left( A_{T}x_{T}^{\prime}+A_{T^{c}}x_{T^{c}}^{\prime}-y\right),z_{T}\rangle\\ &=\langle A_{T}^{\ast}\left( Ax^{\prime}-y\right),z_{T}\rangle\\ &=0 \end{aligned} $$

The last step is due to the feasibility of $x^{\prime }$, i.e., $y=Ax^{\prime }$. The inner product can also be written as

$\langle A\mu ^{\prime }-y,Az\rangle =\langle (A\mu ^{\prime }-Ax-e),Az\rangle =0$.

Therefore,

$\langle (\mu ^{\prime }-x),A^{\ast }Az\rangle =\langle e,Az\rangle ,~~\forall z\in \mathbb {C}^{N}$ supported on T.

Since $(\mu ^{\prime }-x)_{T}$ is supported on T, one has

$\langle (\mu ^{\prime }-x),A^{\ast }A(\mu ^{\prime }-x)_{T}\rangle =\langle e,A(\mu ^{\prime }-x)_{T}\rangle .$

Consequently,

$$ \begin{aligned} \|\left( \mu^{\prime}-x\right)_{T}\|_{2}^{2}&=\langle\left( \mu^{\prime}-x\right),\left( \mu^{\prime}-x\right)_{T}\rangle\\ &=|\langle \left( x-\mu^{\prime}\right),\left( I-A^{\ast}A\right)\left( x-\mu^{\prime}\right)_{T}\rangle+\langle e,A\left( \mu^{\prime}-x\right)_{T}\rangle|\\ &\leq\delta_{s+t}\|x-\mu^{\prime}\|_{2}\|(x-\mu^{\prime})_{T}\|_{2}+\sqrt{1+\delta_{t}}\|e\|_{2}\|\left( x-\mu^{\prime}\right)_{T}\|_{2} \end{aligned} $$

Using Lemma 7.2, Cauchy-Schwarz inequality and Definition 7.1 can obtain the last inequality. Therefore, we have

$\|(x-\mu ^{\prime })_{T}\|_{2}\leq \delta _{s+t}\|x-\mu ^{\prime }\|_{2}+\sqrt {1+\delta _{t}}\|e\|_{2}$.

It then follows that

$$ \begin{aligned} \|(x-\mu^{\prime})\|_{2}^{2}&=\|(x-\mu^{\prime})_{T}\|_{2}^{2}+\|(x-\mu^{\prime})_{T^{c}}\|_{2}^{2}\\ &\leq\left( \delta_{s+t}\|x-\mu^{\prime}\|_{2}+\sqrt{1+\delta_{t}}\|e\|_{2}\right)^{2}+\|x_{T^{c}}\|_{2}^{2}. \end{aligned} $$

In other words,

$\left (\sqrt {1-\delta _{s+t}^{2}}\|(x-\mu ^{\prime })\|_{2}-\frac {\delta _{s+t}\sqrt {1+\delta _{t}}}{\sqrt {1-\delta _{s+t}^{2}}}\|e\|_{2}\right )^{2} \leq \frac {1+\delta _{t}}{1-\delta _{s+t}^{2}}\|e\|_{2}^{2}+\|x_{T^{c}}\|_{2}^{2}.$

It means that

$$ \begin{aligned} \|(x-\mu^{\prime})\|_{2}&\leq\frac{\delta_{s+t}\sqrt{1+\delta_{t}}\|e\|_{2}+\sqrt{(1+\delta_{t})\|e\|_{2}^{2}+({1-\delta_{s+t}^{2})\|x_{T^{c}}\|_{2}^{2}}}}{1-\delta_{s+t}^{2}}\\ &\leq\frac{\|x_{T^{c}}\|_{2}}{\sqrt{1-\delta_{s+t}^{2}}}+\frac{\sqrt{1+\delta_{t}}\|e\|_{2}}{1-\delta_{s+t}}. \end{aligned} $$

□

1.4 Proof of Lemma 4.1.

Proof

As defined, $\widehat {x}\in \mathbb {C}^{N}_{+}$ is the nonincreasing rearrangement of a vector $x=(x_{1}, x_{2},\cdots ,x_{N})\in \mathbb {C}^{N}$, i.e., $\widehat {x}_{1}\geq \widehat {x}_{2}\geq {\ldots } \widehat {x}_{N}\geq 0$, and there exists a permutation π of {1,…,N} such that $\widehat {x}_{i}=|x_{\pi (i)}|$ for all i ∈{1,…,N}. For NST+HT+FB, the hypothesis is $\pi (\{1,\ldots ,p\})\subseteq T_{k}$ and the goal is to prove that $\pi (\{1,\ldots ,p+q\})\subseteq T_{k+\ell }$. That is to say the $|\left (\mu ^{\ell +k-1}+A^{\ast }\left (AA^{\ast }\right )^{-1}\left (y-A\mu ^{\ell +k-1}\right )\right )_{\pi (j)}|$ for j ∈{1,…p + q} are among the s largest entries of $|\left (\mu ^{\ell +k-1}+A^{\ast }\left (AA^{\ast }\right )^{-1}\left (y-A\mu ^{\ell +k-1}\right )\right )_{i}|$ for i ∈{1,…,N}. Since supp(x) = S and |S| = s, it is then enough to prove that

$$ \begin{aligned} &\min \limits_{j\in\{1,\ldots,p+q\}} |\left( \mu^{\ell+k-1}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( y-A\mu^{\ell+k-1}\right)\right)_{\pi(j)}|\\ >&\max \limits_{i\in S^{c}} |\left( \mu^{\ell+k-1}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( y-A\mu^{\ell+k-1}\right)\right)_{i}|. \end{aligned} $$

(1)

For j ∈{1,…p + q} and i ∈ S^c, in view of

$$ \begin{aligned} &|\left( \mu^{\ell+k-1}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( y-A\mu^{\ell+k-1}\right)\right)_{\pi(j)}|\\ \geq&|x_{\pi(j)}|-|\left( -x+\mu^{\ell+k-1}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( y-A\mu^{\ell+k-1}\right)\right)_{\pi(j)}|\\ \geq& \widehat{x}_{p+q}-|\left( \left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\ell+k-1}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e\right)_{\pi(j)}|,\\ &|\left( \mu^{\ell+k-1}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( y-A\mu^{\ell+k-1}\right)\right)_{i}|\\ =&|\left( -x+\mu^{\ell+k-1}+A^{\ast}\left( AA^{\ast}\right)^{-1}\left( y-A\mu^{\ell+k-1}\right)\right)_{i}|\\ =&|\left( \left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\ell+k-1}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e\right)_{i}|, \end{aligned} $$

For the proof of (1), one only needs to prove next, for all j ∈{1,…,p + q} and i ∈ S^c,

$$ \begin{aligned} \widehat{x}_{p+q}>&|\left( \left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\ell+k-1}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e\right)_{\pi(j)}|\\ &+|\left( \left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\ell+k-1}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e\right)_{i}|. \end{aligned} $$

(2)

The right-hand side can be bounded by

$$ \begin{aligned} &\sqrt{2}\|\left( \left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\ell+k-1}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e\right)_{\{\pi(j),i\}}\|_{2}\\ \leq&\sqrt{2}\|\left( \left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\ell+k-1}-x\right)\right)_{\{\pi(j),i\}}\|_{2}\\&+\sqrt{2}\|\left( A^{\ast}\left( AA^{\ast}\right)^{-1}e\right)_{\{\pi(j),i\}}\|_{2}\\ \leq&\sqrt{2}\gamma_{2s+2}\|\mu^{\ell+k-1}-x\|_{2}+\sqrt{2(1+\theta_{2})}\|e\|_{2}\\ \leq&\sqrt{2}\gamma_{2s+2}\left( \rho_{3s}^{\ell-1}\|u^{k}-x\|_{2}+\frac{\tau_{2s}(1-\rho_{3s}^{\ell-1})\|e\|_{2}}{1-\rho_{3s}}\right)+\sqrt{2(1+\theta_{2})}\|e\|_{2}, \end{aligned} $$

where Remark 3.1 was used ℓ − 1 times in the last step. Furthermore, considering Lemma 3.2 and the assumption that $\pi (\{1,\ldots ,p\})\subseteq T_{k}$, we have

$$ \begin{aligned} &\sqrt{2}\|[\left( I-A^{\ast}\left( AA^{\ast}\right)^{-1}A\right)\left( \mu^{\ell+k-1}-x\right)+A^{\ast}\left( AA^{\ast}\right)^{-1}e]_{\{\pi(j),i\}}\|_{2}\\ \leq&\sqrt{2}\gamma_{2s+2}\rho_{3s}^{\ell-1}\left( \frac{1 }{\sqrt{\left( 1-\delta_{2s}^{2}\right)}}\|x_{{T^{c}_{k}}}\|_{2}+\frac{\sqrt{1+\delta_{s}}}{1-\delta_{2s}}\|e\|_{2}\right)\\&+\frac{\sqrt{2}\gamma_{2s+2}\tau_{2s}\left( 1-\rho_{3s}^{\ell-1}\right)\|e\|_{2}}{1-\rho_{3s}}+\sqrt{2\left( 1+\theta_{2}\right)}\|e\|_{2} \end{aligned} $$

$$ \begin{aligned} \leq&\rho_{3s}^{\ell}\|x_{{T^{c}_{k}}}\|_{2}+\left( \sqrt{2}\gamma_{2s+2}\rho_{3s}^{\ell-1}\frac{\sqrt{1+\delta_{s}}}{1-\delta_{2s}}\right.\\&\left.+\frac{\sqrt{2}\gamma_{2s+2}\tau_{2s}(1-\rho_{3s}^{\ell-1})}{1-\rho_{3s}}+\sqrt{2(1+\theta_{2})}\right)\|e\|_{2}\\ \leq&\rho_{3s}^{\ell}\|x_{\pi(\{1,\ldots,p\})^{c}}\|_{2}+\left( \sqrt{2}\gamma_{3s}\frac{\sqrt{1+\delta_{3s}}}{1-\delta_{3s}}+\frac{\sqrt{2}\gamma_{3s}\tau_{3s}}{1-\rho_{3s}}+\sqrt{2(1+\theta_{3s})}\right)\|e\|_{2}\\ \leq&\rho_{3s}^{\ell}\|\widehat{x}_{\{p+1,\ldots,s\}}\|_{2}+\left( \sqrt{2}\gamma_{3s}\frac{\sqrt{1+\delta_{3s}}}{1-\delta_{3s}}+\frac{\sqrt{2}\gamma_{3s}\tau_{3s}}{1-\rho_{3s}}+\sqrt{2(1+\theta_{3s})}\right)\|e\|_{2}\\ \leq&\rho_{3s}^{\ell}\|\widehat{x}_{\{p+1,\ldots,s\}}\|_{2}+\omega_{3s}\|e\|_{2}. \end{aligned} $$

Therefore, if the condition (4.1) holds, then (2) is proved. □