Near optimal bound of orthogonal matching pursuit using restricted isometric constant

Abstract

As a paradigm for reconstructing sparse signals using a set of under sampled measurements, compressed sensing has received much attention in recent years. In identifying the sufficient condition under which the perfect recovery of sparse signals is ensured, a property of the sensing matrix referred to as the restricted isometry property (RIP) is popularly employed. In this article, we propose the RIP based bound of the orthogonal matching pursuit (OMP) algorithm guaranteeing the exact reconstruction of sparse signals. Our proof is built on an observation that the general step of the OMP process is in essence the same as the initial step in the sense that the residual is considered as a new measurement preserving the sparsity level of an input vector. Our main conclusion is that if the restricted isometry constant δ _K of the sensing matrix satisfies

$${\delta }_K<\frac{\sqrt{K-1}}{\sqrt{K-1+K}}$$

then the OMP algorithm can perfectly recover K(> 1)-sparse signals from measurements. We show that our bound is sharp and indeed close to the limit conjectured by Dai and Milenkovic.

1 Introduction

As a paradigm to acquire sparse signals at a rate significantly below Nyquist rate, compressive sensing has received much attention in recent years [1–17]. The goal of compressive sensing is to recover the sparse vector using small number of linearly transformed measurements. The process of acquiring compressed measurements is referred to as sensing while that of recovering the original sparse signals from compressed measurements is called reconstruction.

In the sensing operation, K-sparse signal vector x, i.e., n-dimensional vector having at most K non-zero elements, is transformed into m-dimensional signal (measurements) y via a matrix multiplication with Φ. This process is expressed as

$$y=\Phi x.$$

(1)

Since n > m for most of the compressive sensing scenarios, the system in (1) can be classified as an underdetermined system having more unknowns than observations, and hence, one cannot accurately solve this inverse problem in general. However, due to the prior knowledge of sparsity information, one can reconstruct x perfectly via properly designed reconstruction algorithms. Overall, commonly used reconstruction strategies in the literature can be classified into two categories. The first class is linear programming (LP) techniques including ℓ₁-minimization and its variants. Donoho [10] and Candes [13] showed that accurate recovery of the sparse vector x from measurements y is possible using ℓ₁-minimization technique if the sensing matrix Φ satisfies restricted isometry property (RIP) with a constant parameter called restricted isometry constant. For each positive integer K, the restricted isometric constant δ _K of a matrix Φ is defined as the smallest number satisfying

$$\left(1-{\delta }_K\right){∥x∥}_2^2\le {∥\Phi x∥}_2^2\le \left(1+{\delta }_K\right){∥x∥}_2^2$$

(2)

for all K-sparse vectors x. It has been shown that if ${\delta }_{2K}<\sqrt{2}-1$[13], the ℓ₁-minimization is guaranteed to recover K-sparse signals exactly.

The second class is greedy search algorithms identifying the support (position of nonzero element) of the sparse signal sequentially. In each iteration of these algorithms, correlations between each column of Φ and the modified measurement (residual) are compared and the index (indices) of one or multiple columns that are most strongly correlated with the residual is identified as the support. In general, the computational complexity of greedy algorithms is much smaller than the LP based techniques, in particular for the highly sparse signals (signals with small K). Algorithms contained in this category include orthogonal matching pursuit (OMP) [1], regularized OMP (ROMP) [18], stagewise OMP (DL Donoho, I Drori, Y Tsaig, JL Starck: Sparse solution of underdetermined linear equations by stagewise orthogonal matching pursuit, submittd), and compressive sampling matching pursuit (CoSaMP) [16].

As a canonical method in this family, the OMP algorithm has received special attention due to its simplicity and competitive reconstruction performance. As shown in our Table, the OMP algorithm performs the support identification followed by the residual update in each iteration and these operations are repeated usually K times. It has been shown that the OMP algorithm is robust in recovering both sparse and near-sparse signals [13] with O(nmK) complexity [1]. Over the years, many efforts have been made to find out the condition (upper bound of restricted isometric constant) guaranteeing the exact recovery of sparse signals. For example, δ_3K< 0.165 for the subspace pursuit [19], δ_4K< 0.1 for the CoSaMP [16], and ${\delta }_{4K}<0.01/\sqrt{\text{log}K}$ for the ROMP [18]. The condition for the OMP is given by [20]

$${\delta }_{K+1}<\frac{1}{3\sqrt{K}}.$$

(3)

Recently, improved conditions of the OMP have been reported including ${\delta }_{K+1}<1/\sqrt{2K}$[21] and ${\delta }_{K+1}<1/\left(\sqrt{K}+1\right)$ (J Wang, B Shim: On recovery limit of orthogonal matching pursuit using restricted isometric property, submitted).

The primary goal of this article is to provide an improved condition ensuring the exact recovery of K-sparse signals of the OMP algorithm. While previously proposed recovery conditions are expressed in terms of δ_{K+ 1}[20, 21], our result, formally described in Theorem 1.1, is expressed in terms of the restricted isometric constant δ _K of order K so that it is perhaps most natural and simple to interpret. For instance, our result together with the Johnson-Lindenstrauss lemma [22] can be used to estimate the compression ratio (i.e., minimal number of measurements m ensuring perfect recovery) when the elements of Φ are chosen randomly [17]. Besides, we show that our result is sharp in the sense that the condition is close to the limit of the OMP algorithm conjectured by Dai and Milenkovic [19], in particular when K is large. Our result is formally described in the following theorem.

Theorem 1.1 (Bound of restricted isometry constant). Suppose x ∈ ℝⁿis a K-sparse signal (K > 1), then the OMP algorithm recovers x from y = Φx ∈ ℝ^mif

$${\delta }_K<\frac{\sqrt{K-1}}{\sqrt{K-1}+K}.$$

(4)

Loosely speaking, since $K/\sqrt{K-1}\approx \sqrt{K}$ for K ≫ 1, the proposed condition approximates to ${\delta }_K<1/\left(1+\sqrt{K}\right)$ for a large K. In order to get an idea how close the proposed bound is from the limit conjectured by Dai and Milenkovic $\left({\delta }_{K+1}=1/\sqrt{K}\right)$, we plot the bound as a function of the sparsity level K in Figure 1. Although the proposed bound is expressed in terms of δ _K while (3) and the limit of Dai and Milenkovic are expressed in terms of δ_{K+ 1}so that the comparison is slightly unfavorable for the proposed bound, we still see that the proposed bound is fairly close to the limit for large K. ^a

Figure 1

Bounds of restricted isometry constant. Note that the proposed bound is expressed in terms of δ _K while (3) and the limit of Dai and Milenkovic are expressed in terms of δ_K+1.

As mentioned, another interesting result we can deduce from Theorem 1.1 is that we can estimate the maximal compression ratio when Gaussian random matrix is employed in the sensing process. Note that the direct investigation of the condition δ _K < ϵ for a given sensing matrix Φ is undesirable, especially when n is large and K is nontrivial, since the extremal singular values of $\left({}_K^n\right)$ sub-matrices need to be tested.

In an alternative way, a condition derived from Johnson-Lindenstrauss lemma has been popularly considered. In fact, it is now well known that m × n random matrices with i.i.d. entries from the Gaussian distribution $\mathcal{N}\left(0,1/m\right)$ obey the RIP with δ _K ≤ ϵ with overwhelming probability if the dimension of the measurements satisfies [17]

$$m\ge \frac{C\cdot K\text{log}\frac{n}{K}}{{\epsilon }^2}$$

(5)

where C is a constant. By applying the result in Theorem 1.1, we can obtain the minimum dimension of m ensuring exact reconstruction of K-sparse signal using the OMP algorithm. Specifically, plugging $\epsilon =\sqrt{K-1}/\left(\sqrt{K-1}+K\right)$ into (5), we get

$$m\ge C\cdot \left(\frac{K^3}{K-1}+\frac{2K^2}{\sqrt{K-1}}+K\right)\text{log}\frac{n}{K}.$$

(6)

This result [m is expressed in the order of $O\left(K^2\text{log}\frac{n}{K}\right)$ is desirable, since the size of measurements m grows moderately with the sparsity level K.

2 Proof of theorem 1.1

2.1 Notations

We now provide a brief summary of the notations used throughout the article.

• T = supp(x) = { i | x _i ≠ 0} is the set of non-zero positions in x.

• |D| is the cardinality of D.

• T\D is the set of all elements contained in T but not in D.

• Φ _D ∈ ℝ^m×|D|is a submatrix of Φ that only contains columns indexed by D.

• x _D ∈ ℝ^|D|is a restriction of the vector x to the elements with indices in D.

• span(Φ _D ) is the span (range) of columns in Φ _D .

• ${\Phi \prime }_D$ denotes the transpose of the matrix Φ _D .

• ${\Phi }_D^{†}={\left({\Phi \prime }_D{\Phi }_D\right)}^{-1}{\Phi \prime }_D$ is the pseudoinverse of Φ _D .

• $P_D={\Phi }_D{\Phi }_D^{†}$ denotes the orthogonal projection onto span(Φ _D ).

• $P_D^{\perp }=I-P_D$ is the orthogonal projection onto the orthogonal complement of span(Φ _D ).

2.2 Preliminaries--definitions and lemmas

In this subsection, we provide useful definition and lemmas used for the proof of Theorem 1.1.

Definition 1 (Restricted orthogonality constant [23]). For two positive integers K and K', if K + K' ≤ n, then K, K'-restricted orthogonality constant θ _K,K' , is the smallest number that satisfies

$$\left|⟨\Phi x,\Phi x\prime ⟩\right|\le {\theta }_{K,K\prime }\parallel x{\parallel }_2\parallel x\prime {\parallel }_2$$

(7)

for all x and x' such that x and x' are K-sparse and K'-sparse respectively, and have disjoint supports.

Lemma 2.1. In the OMP algorithm, the residual r^kis orthogonal to the columns selected in previous iterations. That is,

$$⟨{\phi }_i,r^k⟩=0$$

(8)

for i ∈ T^k.

Lemma 2.2 (Monotonicity of δ _K [19]). If the sensing matrix satisfies the RIP of orders K₁and K₂, respectively, then

$${\delta }_{K_1}\le {\delta }_{K_2}$$

for any K₁ ≤ K₂. This property is referred to as the monotonicity of the restricted isometric constant.

Lemma 2.3 (A direct consequence of RIP [19]). Let I ⊂ {1,2,... ,n} and Φ _I be the sub-matrix of Φ that contains columns of Φ indexed by I. If δ_|I|< 1, then for any u ∈ ℝ^|I|,

$$\left(1-{\delta }_{\left|I\right|}\right)\parallel u{\parallel }_2\le \parallel {\Phi \prime }_I{\Phi }_{I}u{\parallel }_2\le \left(1+{\delta }_{\left|I\right|}\right)\parallel u{\parallel }_2.$$

Lemma 2.4 (Square root lifting inequality [23]). For α ≥ 1 and positive integers K, K' such that αK' is also an integer, we have

$${\theta }_{K,\alpha K\prime }\le \sqrt{\alpha }{\theta }_{K,K\prime }.$$

(9)

Lemma 2.5 (Lemma 2.1 in [13]). For all x, x' ∈ ℝⁿsupported on disjoint subsets I₁, I₂ ⊂ {1, 2,..., n}, we have

$$\left|⟨\Phi x,\Phi x\prime ⟩\right|\le {\delta }_{\left|I_1\right|+\left|I_2\right|}\parallel x{\parallel }_2\parallel x\prime {\parallel }_2.$$

Lemma 2.6. For two disjoint sets I₁, I₂ ⊂ {1, 2,... ,n}, let${\theta }_{\left|I_1\right|,\left|I_2\right|}$be the |I₁|, |I₂|-restricted orthogonality constant of Φ. If |I₁| + |I₂| ≤ n, then

$$\parallel {\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}{\parallel }_2\le {\theta }_{\left|I_1\right|,\left|I_2\right|}\parallel x{\parallel }_2.$$

(10)

Proof. Let $u\in {ℝ}^{\left|I_1\right|}$ be a unit vector, then we have

$$\underset{u:\parallel u{\parallel }_2=1}{\text{max}}\parallel u\prime \left({\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}\right){\parallel }_2=\parallel {\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}{\parallel }_2.$$

(11)

where the maximum of inner product is achieved when u is in the same direction of ${\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}$ i.e., $\left(u={\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}/\parallel {\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}{\parallel }_2\right)$. Moreover, from Definition 1, we have

$$\begin{array}{ll}\hfill \parallel u\prime {\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}{\parallel }_2& =\left|⟨{\Phi }_{I_1}u,{\Phi }_{I_2}{x}_{I_2}⟩\right|\\ \le {\theta }_{\left|I_1\right|,\left|I_2\right|}\parallel u{\parallel }_2\parallel x{\parallel }_2\\ ={\theta }_{\left|I_1\right|,\left|I_2\right|}\parallel x{\parallel }_2\end{array}$$

(12)

and thus

$$\parallel {\Phi \prime }_{I_1}{\Phi }_{I_2}{x}_{I_2}{\parallel }_2\le {\theta }_{\left|I_1\right|,\left|I_2\right|}\parallel x{\parallel }_2.$$

Lemma 2.7. For two disjoint sets I₁,I₂with |I₁| + |I₂| ≤ n, we have

$${\delta }_{\left|I_1\right|+\left|I_2\right|}\ge {\theta }_{\left|I_1\right|,\left|I_2\right|}.$$

(13)

Proof. From Lemma 2.5 we directly have

$$\left|⟨{\Phi }_{I_1}{x}_{I_1},{\Phi }_{I_2}{x}_{I_2}⟩\right|\le {\delta }_{\left|I_1\right|+\left|I_2\right|}{∥x_{I_1}∥}_2{∥x_{I_2}∥}_2.$$

(14)

By Definition 1, ${\theta }_{\left|I_1\right|,\left|I_2\right|}$ is the minimal value satisfying

$$\left|⟨{\Phi }_{I_1}{x}_{I_1},{\Phi }_{I_2}{x}_{I_2}⟩\right|\le {\theta }_{\left|I_1\right|,\left|I_2\right|}\parallel x_{I_1}{\parallel }_2\parallel x_{I_2}{\parallel }_2,$$

(15)

and this completes the proof of the lemma.

2.3 Proof of theorem 1.1

Now we turn to the proof of our main theorem. Our proof is in essence based on the mathematical induction; First, we show that the index t¹ found at the first iteration is correct (t¹ ∈ T) under (4) and then we show that t^{k+ 1}is also correct (more accurately T^k= {t¹,t², ...,t^k} ∈ T then t^k+1∈ T\T^k) under (4).

Proof. Let t^kbe the index of the column maximally correlated with the residual r^k-1in the k-th iteration of the OMP algorithm. Since r^k-1= y for k = 1, t¹ can be expressed as

$$t^1=\text{arg}\underset{i}{\text{max}}\left|⟨{\phi }_i,y⟩\right|$$

(16)

and also

$$\left|⟨{\phi }_{t^1},y⟩\right|=\underset{i}{\text{max}}\left|⟨{\phi }_{i,}y⟩\right|$$

(17)

$$\ge \sqrt{\frac{1}{\left|T\right|}\sum _{j\in T}{\left|⟨{\phi }_j,y⟩\right|}^2}$$

(18)

$$=\frac{1}{\sqrt{K}}\parallel {\Phi \prime }_{T}y{\parallel }_2$$

(19)

where (19) uses the fact |T| = K (x is K-sparse supported on T). Now that y = Φ _T x _T , we have

$$\left|⟨{\phi }_{t^1},y⟩\right|\ge \frac{1}{\sqrt{K}}\parallel {\Phi \prime }_T{\Phi }_T{x}_T{\parallel }_2$$

(20)

$$\ge \frac{1}{\sqrt{K}}\left(1-{\delta }_K\right)\parallel x_T{\parallel }_2$$

(21)

where (21) follows from Lemma 2.3.

Now, suppose that t¹ does not belong to the support of x (i.e., t¹ ∉ T), then

$$\left|⟨{\phi }_{t^1},y⟩\right|=\parallel \phi {\prime }_{t^1}{\Phi }_T{x}_T{\parallel }_2$$

(22)

$$\le {\theta }_{1,K}\parallel x_T{\parallel }_2$$

(23)

where (23) is from Lemma 2.6. This case, however, will never occur if

$$\frac{1}{\sqrt{K}}\left(1-{\delta }_K\right)\parallel x_T{\parallel }_2>{\theta }_{1,K}\parallel x_T{\parallel }_2$$

(24)

or

$$\sqrt{K}{\theta }_{1,K}+{\delta }_K<1.$$

(25)

Let α = K/(K - 1), then α(K - 1) = K is an integer and

$${\theta }_{1,K}={\theta }_{1,\alpha \left(K-1\right)}$$

(26)

$$\le \sqrt{\alpha }{\theta }_{1,K-1}$$

(27)

$$\le \sqrt{\frac{K}{K-1}}{\delta }_K$$

(28)

where (27) and (28) follow from Lemma 2.4 and 3.1, respectively. Thus, (25) holds true when

$$\sqrt{K}\sqrt{\frac{K}{K-1}}{\delta }_K+{\delta }_K<1,$$

which yields

$${\delta }_K<\frac{\sqrt{K-1}}{\sqrt{K-1}+K}.$$

(29)

In summary, if ${\delta }_K<\sqrt{K-1}/\left(\sqrt{K-1}+K\right)$, then t¹ ∈ T for the first iteration of the OMP algorithm. Now we assume that former k iterations are successful (T^k= {t¹, t²,... ,t^k} ∈ T) for 1 ≤ k ≤ K - 1. Then it suffices to show that t^k+1is in T but not in T^k(i.e., t^k+1∈ T\T^k). Recall from Table 1 that the residual at the k-th iteration of the OMP is expressed as

Table 1 OMP algorithm

$$r^k=y-{\Phi }_{T^k}{\widehat{x}}_{T^k}.$$

(30)

Since y = Φ _T x _T and ${\Phi }_{T^k}$ is a submatrix of Φ _T , r^k∈ span (Φ _T ) and hence r^kcan be expressed as a linear combination of the |T| (= K) columns of Φ _T . Accordingly, we can express r^kas r^k= Φx^k where the support (set of indices for nonzero elements) of x^kis contained in the support of x. That is, r^kis a measurement of K-sparse signal x^k using the sensing matrix Φ.

Therefore, it is clear that if T^k∈ T, then t^k+1∈ T under (29). Recalling that the residual r^kis orthogonal to the column already selected (〈φ _i , r^k〉 = 0 for i ∈ T^k) from Lemma 1, index of these columns is not selected again (see the identify step in Table 1) and hence t^k+1∈T\T^k. This indicates that under the condition in (4) all the indices in the support T will be identified within K iterations (i.e., T^K= T) and therefore

$${\widehat{x}}_{T^K}=\text{arg}\underset{x}{\text{min}}\parallel y-{\Phi }_{T^K}x{\parallel }_2$$

(31)

$$={\Phi }_{T^K}^{†}y$$

(32)

$$={\Phi }_T^{†}y$$

(33)

$$={\left({\Phi }_T^t{\Phi }_T\right)}^{-1}{\Phi \prime }_T{\Phi }_T{x}_T$$

(34)

$$=x_T,$$

(35)

which completes the proof.

3 Discussions

In [19], Dai and Milenkovic conjectured that the sufficient condition of the OMP algorithm guaranteeing exact recovery of K-sparse vector cannot be further relaxed to ${\delta }_{K+1}=1/\sqrt{K}$. This conjecture says that if the RIP condition is given by δ_{K+ 1}< ϵ then ϵ should be strictly smaller than $1/\sqrt{K}$. In [20], this conjecture has been confirmed via experiments for K = 2.

We now show that our result in Theorem 1.1 agrees with the conjecture, leaving only marginal gap from the limit. Note that since we cannot directly compare Dai and Milenkovic's conjecture (expressed in term of δ_{K+ 1}) with our condition (expressed in term of δ _K ), we need to modify our result. Following proposition provides a bit loose bound (sufficient condition) of our result expressed in the form of ${\delta }_{K+1}\le 1/\left(\sqrt{K}+\theta \right)$.

Proposition 3.1. If${\delta }_{K+1}<1/\left(\sqrt{K}+3-\sqrt{2}\right)$then${\delta }_K<\sqrt{K-1}/\left(\sqrt{K-1}+K\right)$.

Proof. Since the inequality

$$\frac{1}{\sqrt{K}+3-\sqrt{2}}\le \frac{\sqrt{K-1}}{\sqrt{K-1}+K}$$

(36)

holds true for any integer K > 1 (see Appendix), if ${\delta }_{K+1}<1/\left(\sqrt{K}+3-\sqrt{2}\right)$ then ${\delta }_{K+1}<\sqrt{K-1}/\left(\sqrt{K-1}+K\right)$. Also, from the monotonicity of the RIP constant (δ _K ≤ δ_K+1), if ${\delta }_{K+1}<\sqrt{K-1}/\left(\sqrt{K-1}+K\right)$ then ${\delta }_K<\sqrt{K-1}/\left(\sqrt{K-1}+K\right)$. Syllogism of above two conditions yields the desired result.

One can clearly observe that ${\delta }_{K+1}<1/\left(\sqrt{K}+3-\sqrt{2}\right)\approx 1/\left(\sqrt{K}+1.5858\right)$ is better than the condition ${\delta }_{K+1}<1/\left(3\sqrt{K}\right)$[20], similar to the result of Wang and Shim, and also close to the achievable limit $\left({\delta }_{k+1}<1/\sqrt{K}\right)$, in particular for large K. Considering that the derived condition ${\delta }_{K+1}<1/\left(\sqrt{K}+3-\sqrt{2}\right)$ is slightly worse than our original condition ${\delta }_K<\sqrt{K-1}/\left(\sqrt{K-1}+K\right)$, we may conclude that our result is fairly close to the optimal.

4 Conclusion

In this article, we have investigated the sufficient condition ensuring exact reconstruction of sparse signal for the OMP algorithm. We showed that if the restricted isometry constant δ _K of the sensing matrix satisfies

$${\delta }_K<\frac{\sqrt{K-1}}{\sqrt{K-1}+K}$$

then the OMP algorithm can perfectly recover K-sparse signals from measurements. Our result directly indicates that the set of sensing matrices for which exact recovery of sparse signal is possible using the OMP algorithm is wider than what has been proved thus far. Another interesting point that we can draw from our result is that the size of measurements (compressed signal) required for the reconstruction of sparse signal grows moderately with the sparsity level.

Appendix--proof of (36)

After some algebra, one can show that (36) can be rewritten as

$$1+\frac{K}{\sqrt{K-1}}-\sqrt{K}\le 3-\sqrt{2}.$$

(37)

Let $f\left(K\right)=1+K/\sqrt{K-1}-\sqrt{K}$ then $f\left(2\right)=3-\sqrt{2}$. Hence, it suffices to show that f(K) is a decreasing function in K ≥ 2 (i.e., f(2) is the maximum for K ≥ 2). In fact, since

$$f^{\prime }\left(K\right)=\frac{\left(K-2\right)\sqrt{K}-\left(K-1\right)\sqrt{K-1}}{2\sqrt{K\left(K-1\right)}\left(K-1\right)},$$

(38)

with

$$\sqrt{K\left(K-1\right)}\left(K-1\right)>0$$

(39)

and

$$\left(K-2\right)\sqrt{K}-\left(K-1\right)\sqrt{K-1}<0$$

(40)

for K ≥ 2, f'(K) < 0 for K ≥ 2, which completes the proof of (36).

Endnote

^aIn Section 3, we provide more rigorous discussions on this issue.