The Road to Deterministic Matrices with the Restricted Isometry Property

Abstract

The restricted isometry property (RIP) is a well-known matrix condition that provides state-of-the-art reconstruction guarantees for compressed sensing. While random matrices are known to satisfy this property with high probability, deterministic constructions have found less success. In this paper, we consider various techniques for demonstrating RIP deterministically, some popular and some novel, and we evaluate their performance. In evaluating some techniques, we apply random matrix theory and inadvertently find a simple alternative proof that certain random matrices are RIP. Later, we propose a particular class of matrices as candidates for being RIP, namely, equiangular tight frames (ETFs). Using the known correspondence between real ETFs and strongly regular graphs, we investigate certain combinatorial implications of a real ETF being RIP. Specifically, we give probabilistic intuition for a new bound on the clique number of Paley graphs of prime order, and we conjecture that the corresponding ETFs are RIP in a manner similar to random matrices.

This is a preview of subscription content, log in to check access.

References

  1. 1.

    Alexeev, B., Cahill, J., Mixon, D.G.: Full spark frames. J. Fourier Anal. Appl. 18(6), 1167–1194 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  2. 2.

    Alon, N., Naor, A.: Approximating the cut-norm via Grothendieck’s inequality. SIAM J. Comput. 35, 787–803 (2006)

    MathSciNet  Article  MATH  Google Scholar 

  3. 3.

    Applebaum, L., Howard, S.D., Searle, S., Calderbank, R.: Chirp sensing codes: deterministic compressed sensing measurements for fast recovery. Appl. Comput. Harmon. Anal. 26, 283–290 (2009)

    MathSciNet  Article  MATH  Google Scholar 

  4. 4.

    Bandeira, A.S., Dobriban, E., Mixon, D.G., Sawin, W.F.: Certifying the restricted isometry property is hard. IEEE Trans. Inf. Theory 59, 3448–3450 (2013)

    MathSciNet  Article  Google Scholar 

  5. 5.

    Baraniuk, R., Davenport, M., DeVore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28, 253–263 (2008)

    MathSciNet  Article  MATH  Google Scholar 

  6. 6.

    Bernstein, S.N.: Theory of Probability, 4th edn. Gostechizdat, Moscow-Leningrad (1946)

    Google Scholar 

  7. 7.

    Bollobás, B.: Random Graphs, 2nd edn. Cambridge Univ. Press, Cambridge (2001)

    Google Scholar 

  8. 8.

    Bourgain, J., Dilworth, S., Ford, K., Konyagin, S., Kutzarova, D.: Explicit constructions of RIP matrices and related problems. Duke Math. J. 159, 145–185 (2011)

    MathSciNet  Article  MATH  Google Scholar 

  9. 9.

    Candès, E.J.: The restricted isometry property and its implications for compressed sensing. C. R. Acad. Sci. Paris, Ser. I 346, 589–592 (2008)

    Article  MATH  Google Scholar 

  10. 10.

    Candès, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 44, 4203–4215 (2005)

    Article  Google Scholar 

  11. 11.

    Candès, E.J., Tao, T.: The Dantzig selector: statistical estimation when p is much larger than n. Ann. Stat. 35, 2313–2351 (2007)

    Article  MATH  Google Scholar 

  12. 12.

    Chung, F.R.K., Graham, R.L., Wilson, R.M.: Quasi-random graphs. Combinatorica 9, 345–362 (1989)

    MathSciNet  Article  MATH  Google Scholar 

  13. 13.

    Cohen, S.D.: Clique numbers of Paley graphs. Quaest. Math. 11, 225–231 (1988)

    Article  MATH  Google Scholar 

  14. 14.

    Davidson, K.R., Szarek, S.J.: Local operator theory, random matrices and Banach spaces. In: Johnson, W.B., Lindenstrauss, J. (eds.) Handbook in Banach Spaces, vol. I, pp. 317–366. Elsevier, Amsterdam (2001)

    Google Scholar 

  15. 15.

    DeVore, R.A.: Deterministic constructions of compressed sensing matrices. J. Complex. 23, 918–925 (2007)

    MathSciNet  Article  MATH  Google Scholar 

  16. 16.

    Donoho, D.L., Elad, M.: Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization. Proc. Natl. Acad. Sci. USA 100, 2197–2202 (2003)

    MathSciNet  Article  MATH  Google Scholar 

  17. 17.

    Fickus, M., Mixon, D.G., Tremain, J.C.: Steiner equiangular tight frames. Linear Algebra Appl. 436, 1014–1027 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  18. 18.

    Gerschgorin, S.: Über die Abgrenzung der Eigenwerte einer Matrix. Izv. Akad. Nauk. USSR Otd. Fiz.-Mat. 7, 749–754 (1931)

    Google Scholar 

  19. 19.

    Graham, S.W., Ringrose, C.J.: Lower bounds for least quadratic non-residues. Prog. Math. 85, 269–309 (1990)

    MathSciNet  Google Scholar 

  20. 20.

    Hoory, S., Linial, N., Wigderson, A.: Expander graphs and their applications. Bull. Am. Math. Soc. 43, 439–561 (2006)

    MathSciNet  Article  MATH  Google Scholar 

  21. 21.

    Laurent, B., Massart, P.: Adaptive estimation of a quadratic functional by model selection. Ann. Stat. 28, 1302–1338 (2000)

    MathSciNet  Article  MATH  Google Scholar 

  22. 22.

    Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995)

    MathSciNet  Article  MATH  Google Scholar 

  23. 23.

    Peralta, R.: On the distribution of quadratic residues and nonresidues modulo a prime number. Math. Comput. 58, 433–440 (1992)

    MathSciNet  Article  MATH  Google Scholar 

  24. 24.

    Rauhut, H.: Stability results for random sampling of sparse trigonometric polynomials. IEEE Trans. Inf. Theory 54, 5661–5670 (2008)

    MathSciNet  Article  MATH  Google Scholar 

  25. 25.

    Renes, J.M.: Equiangular tight frames from Paley tournaments. Linear Algebra Appl. 426, 497–501 (2007)

    MathSciNet  Article  MATH  Google Scholar 

  26. 26.

    Rudelson, M., Vershynin, R.: On sparse reconstruction from Fourier and Gaussian measurements. Commun. Pure Appl. Math. 61, 1025–1045 (2008)

    MathSciNet  Article  MATH  Google Scholar 

  27. 27.

    Sachs, H.: Über selbstkomplementäre Graphen. Publ. Math. (Debr.) 9, 270–288 (1962)

    MathSciNet  Google Scholar 

  28. 28.

    Seidel, J.J.: A survey of two-graphs. In: Proc. Intern. Coll. Teorie Combinatorie, pp. 481–511 (1973)

    Google Scholar 

  29. 29.

    Stevenhagen, P., Lenstra, H.W.: Chebotarëv and his density theorem. Math. Intell. 18, 26–37 (1996)

    MathSciNet  Article  MATH  Google Scholar 

  30. 30.

    Strohmer, T., Heath, R.W.: Grassmannian frames with applications to coding and communication. Appl. Comput. Harmon. Anal. 14, 257–275 (2003)

    MathSciNet  Article  MATH  Google Scholar 

  31. 31.

    Tao, T.: Open question: deterministic UUP matrices. http://terrytao.wordpress.com/2007/07/02/open-question-deterministic-uup-matrices

  32. 32.

    Temlyakov, V.: Greedy Approximations. Cambridge University Press, Cambridge (2011)

    Google Scholar 

  33. 33.

    van Lint, J.H., Seidel, J.J.: Equilateral point sets in elliptic geometry. Nederl. Akad. Wetensch. Proc. Ser. A 69, 335–348 (1966). Indag. Math. 28

    MATH  Google Scholar 

  34. 34.

    Waldron, S.: On the construction of equiangular frames from graphs. Linear Algebra Appl. 431, 2228–2242 (2009)

    MathSciNet  Article  MATH  Google Scholar 

  35. 35.

    Welch, L.R.: Lower bounds on the maximum cross correlation of signals. IEEE Trans. Inf. Theory 20, 397–399 (1974)

    MathSciNet  Article  MATH  Google Scholar 

  36. 36.

    Xia, P., Zhou, S., Giannakis, G.B.: Achieving the Welch bound with difference sets. IEEE Trans. Inf. Theory 51, 1900–1907 (2005)

    MathSciNet  Article  MATH  Google Scholar 

  37. 37.

    Yurinskii, V.V.: Exponential inequalities for sums of random vectors. J. Multivar. Anal. 6, 473–499 (1976)

    MathSciNet  Article  MATH  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Afonso S. Bandeira.

Additional information

The authors thank Prof. Peter Sarnak and Joel Moreira for insightful discussions and helpful suggestions. ASB was supported by NSF Grant No. DMS-0914892, MF was supported by NSF Grant No. DMS-1042701 and AFOSR Grant Nos. F1ATA01103J001 and F1ATA00183G003, and DGM was supported by the A.B. Krongard Fellowship. The views expressed in this article are those of the authors and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the US Government.

Communicated by Vladimir Temlyakov.

Appendix

Appendix

In this section, we prove Theorem 13, which states that a matrix with \((K,\hat{\theta})\)-flat restricted orthogonality has \(\theta_{K}\leq C\hat{\theta}\log K\), that is, it has restricted orthogonality. The proof below is adapted from the proof of Lemma 3 in [8]. Our proof has the benefit of being valid for all values of K (as opposed to sufficiently large K in the original [8]), and it has near-optimal constants where appropriate. Moreover in this version, the columns of the matrix are not required to have unit norm.

Proof of Theorem 13

Given arbitrary disjoint subsets \(\mathcal{I},\mathcal{J}\subseteq\{1,\ldots,N\}\) with \(|\mathcal{I}|,|\mathcal{J}|\leq K\), we will bound the following quantity three times, each time with different constraints on \(\{x_{i}\}_{i\in\mathcal{I}}\) and \(\{y_{j}\}_{j\in\mathcal{J}}\):

$$ \bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_i\varphi_i,\sum_{j\in\mathcal{J}}y_j\varphi_j \bigg\rangle\bigg|. $$
(23)

To be clear, our third bound will have no constraints on \(\{x_{i}\}_{i\in\mathcal{I}}\) and \(\{y_{j}\}_{j\in\mathcal{J}}\), thereby demonstrating restricted orthogonality. Note that by assumption, (23) is \(\leq\hat{\theta}(|\mathcal{I}||\mathcal{J}|)^{1/2}\) whenever the x i ’s and y j ’s are in {0,1}. We first show that this bound is preserved when we relax the x i ’s and y j ’s to lie in the interval [0,1].

Pick a disjoint pair of subsets \(\mathcal{I}',\mathcal{J}'\subseteq\{1,\ldots,N\}\) with \(|\mathcal{I}'|,|\mathcal{J}'|\leq K\). Starting with some \(k\in\mathcal{I}'\), note that flat restricted orthogonality gives that

$$\begin{aligned} \bigg|\bigg\langle \sum_{i\in\mathcal{I}}\varphi_i,\sum_{j\in\mathcal{J}}\varphi_j \bigg\rangle\bigg| &\leq\hat{\theta}(|\mathcal{I}||\mathcal{J}|)^{1/2},\\ \bigg|\bigg\langle \sum_{i\in\mathcal{I}\setminus\{k\}}\varphi_i,\sum_{j\in\mathcal{J}}\varphi_j \bigg\rangle\bigg| &\leq\hat{\theta}(|\mathcal{I}\setminus\{k\}||\mathcal{J}|)^{1/2} \leq\hat{\theta}(|\mathcal{I}||\mathcal{J}|)^{1/2} \end{aligned}$$

for every disjoint \(\mathcal{I},\mathcal{J}\subseteq\{1,\ldots,N\}\) with \(|\mathcal{I}|,|\mathcal{J}|\leq K\) and \(k\in\mathcal{I}\). Thus, we may take any x k ∈[0,1] to form a convex combination of these two expressions, and then the triangle inequality gives

$$\begin{aligned} \hat{\theta}(|\mathcal{I}||\mathcal{J}|)^{1/2} &\geq x_k\bigg|\bigg\langle \sum_{i\in\mathcal{I}}\varphi_i,\sum_{j\in\mathcal{J}}\varphi_j \bigg\rangle\bigg|+(1-x_k)\bigg|\bigg\langle \sum_{i\in\mathcal{I}\setminus\{k\}}\varphi_i,\sum_{j\in\mathcal{J}}\varphi_j \bigg\rangle\bigg|\\ &\geq\bigg|x_k\bigg\langle \sum_{i\in\mathcal{I}}\varphi_i,\sum_{j\in\mathcal{J}}\varphi_j \bigg\rangle+(1-x_k)\bigg\langle \sum_{i\in\mathcal{I}\setminus\{k\}}\varphi_i,\sum_{j\in\mathcal{J}}\varphi_j \bigg\rangle\bigg|\\ &=\bigg|\bigg\langle \sum_{i\in\mathcal{I}}\bigg\{\begin{array}{cc}x_k,&i=k\\ 1,&i\neq k\end{array}\bigg\}\varphi_i,\sum_{j\in\mathcal{J}}\varphi_j \bigg\rangle\bigg|. \end{aligned}$$
(24)

Since (24) holds for every disjoint \(\mathcal{I},\mathcal{J}\subseteq\{1,\ldots,N\}\) with \(|\mathcal{I}|,|\mathcal{J}|\leq K\) and \(k\in\mathcal{I}\), we can do the same thing with an additional index \(i\in\mathcal{I}'\) or \(j\in\mathcal{J}'\), and replace the corresponding unit coefficient with some x i or y j in [0,1]. Continuing in this way proves the claim that (23) is \(\leq\hat{\theta}(|\mathcal{I}||\mathcal{J}|)^{1/2}\) whenever the x i ’s and y j ’s lie in the interval [0,1].

For the second bound, we assume the x i ’s and y j ’s are nonnegative with unit norm: \(\sum_{i\in\mathcal{I}}x_{i}^{2}=\sum_{j\in\mathcal{J}}y_{j}^{2}=1\). To bound (23) in this case, we partition \(\mathcal{I}\) and \(\mathcal{J}\) according to the size of the corresponding coefficients:

$$ \mathcal{I}_k:=\{i\in\mathcal{I}:2^{-(k+1)}<x_i\leq 2^{-k}\}, \qquad \mathcal{J}_k:=\{j\in\mathcal{J}:2^{-(k+1)}<y_j\leq 2^{-k}\}. $$

Note the unit-norm constraints ensure that \(\mathcal{I}=\bigcup_{k=0}^{\infty}\mathcal{I}_{k}\) and \(\mathcal{J}=\bigcup_{k=0}^{\infty}\mathcal{J}_{k}\). The triangle inequality thus gives

$$\begin{aligned} \bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_i\varphi_i,\sum_{j\in\mathcal{J}}y_j\varphi_j \bigg\rangle\bigg| &=\bigg|\bigg\langle \sum_{k_1=0}^\infty\sum_{i\in\mathcal{I}_{k_1}}x_i\varphi_i,\sum_{k_2=0}^\infty\sum_{j\in\mathcal{J}_{k_2}}y_j\varphi_j \bigg\rangle\bigg|\\ &\leq\sum_{k_1=0}^\infty\sum_{k_2=0}^\infty 2^{-(k_1+k_2)} \bigg|\bigg\langle \sum_{i\in\mathcal{I}_{k_1}}\frac{x_i}{2^{-k_1}}\varphi_i,\sum_{j\in\mathcal{J}_{k_2}}\frac{y_j}{2^{-k_2}}\varphi_j \bigg\rangle\bigg|. \end{aligned}$$
(25)

By the definitions of \(\mathcal{I}_{k_{1}}\) and \(\mathcal{J}_{k_{2}}\), the coefficients of φ i and φ j in (25) all lie in [0,1]. As such, we continue by applying our first bound:

$$\begin{aligned} \bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_i\varphi_i,\sum_{j\in\mathcal{J}}y_j\varphi_j \bigg\rangle\bigg| &\leq\sum_{k_1=0}^\infty\sum_{k_2=0}^\infty 2^{-(k_1+k_2)} \hat{\theta}(|\mathcal{I}_{k_1}||\mathcal{J}_{k_2}|)^{1/2}\\ &=\hat{\theta}\bigg(\sum_{k=0}^\infty2^{-k}|\mathcal{I}_{k}|^{1/2}\bigg)\bigg(\sum_{k=0}^\infty2^{-k}|\mathcal{J}_{k}|^{1/2}\bigg). \end{aligned}$$
(26)

We now observe from the definition of \(\mathcal{I}_{k}\) that

$$ 1 =\sum_{i\in\mathcal{I}}x_i^2 =\sum_{k=0}^\infty\sum_{i\in\mathcal{I}_k}x_i^2 >\sum_{k=0}^\infty 4^{-(k+1)}|\mathcal{I}_k|. $$

Thus for any positive integer t, the Cauchy-Schwarz inequality gives

$$\begin{aligned} \sum_{k=0}^\infty 2^{-k}|\mathcal{I}_k|^{1/2} &=\sum_{k=0}^{t-1} 2^{-k}|\mathcal{I}_k|^{1/2}+\sum_{k=t}^\infty 2^{-k}|\mathcal{I}_k|^{1/2}\\ &\leq t^{1/2}\bigg(\sum_{k=0}^{t-1} 4^{-k}|\mathcal{I}_k|\bigg)^{1/2}+\sum_{k=t}^\infty 2^{-k}K^{1/2}\\ &< 2(t^{1/2}+K^{1/2}2^{-t}), \end{aligned}$$
(27)

and similarly for the \(\mathcal{J}_{k}\)’s. For a fixed K, we note that (27) is minimized when \(K^{1/2}2^{-t}=\frac{t^{-1/2}}{2\log 2}\), and so we pick t to be the smallest positive integer such that \(K^{1/2}2^{-t}\leq\frac{t^{-1/2}}{2\log 2}\). With this, we continue (26):

$$\begin{aligned} \bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_i\varphi_i,\sum_{j\in\mathcal{J}}y_j\varphi_j \bigg\rangle\bigg| &<\hat{\theta}\big(2(t^{1/2}+K^{1/2}2^{-t})\big)^2\\ &\leq 4\hat{\theta}\bigg(t^{1/2}+\frac{t^{-1/2}}{2\log 2}\bigg)^2 =4\hat{\theta}\bigg(t+\frac{1}{\log 2}+\frac{1}{(2\log 2)^2t}\bigg). \\ \\ \end{aligned}$$
(28)

From here, we claim that \(t\leq\lceil\frac{\log K}{\log 2}\rceil\). Considering the definition of t, this is easily verified for K=2,3,…,7 by showing \(K^{1/2}2^{-s}\leq\frac{s^{-1/2}}{2\log 2}\) for \(s=\lceil\frac{\log K}{\log 2}\rceil\). For K≥8, one can use calculus to verify the second inequality of the following:

$$ K^{1/2}2^{-\lceil\frac{\log K}{\log 2}\rceil} \leq K^{1/2}2^{-\frac{\log K}{\log 2}} \leq \frac{1}{2\log 2}\bigg(\frac{\log K}{\log 2}+1\bigg)^{-1/2} \leq \frac{1}{2\log 2}\bigg\lceil\frac{\log K}{\log 2}\bigg\rceil^{-1/2}, $$

meaning \(t\leq\lceil\frac{\log K}{\log 2}\rceil\). Substituting \(t\leq\frac{\log K}{\log 2}+1\) and t≥1 into (28) then gives

$$ \bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_i\varphi_i,\sum_{j\in\mathcal{J}}y_j\varphi_j \bigg\rangle\bigg| <4\hat{\theta}\bigg(\frac{\log K}{\log2}+1+\frac{1}{\log 2}+\frac{1}{(2\log 2)^2}\bigg)\\ =\hat{\theta}(C_0\log K+C_1), $$

with C 0≈5.77, C 1≈11.85. As such, (23) is \(\leq C'\hat{\theta}\log K\) with \(C'=C_{0}+\frac{C_{1}}{\log 2}\) in this case.

We are now ready for the final bound on (23) in which we apply no constraints on the x i ’s and y j ’s. To do this, we consider the positive and negative real and imaginary parts of these coefficients:

$$ x_i=\sum_{k=0}^3x_{i,k}\mathrm{i}^k\quad \mbox{s.t.}\quad x_{i,k}\geq0\quad \forall k, $$

and similarly for the y j ’s. With this decomposition, we apply the triangle inequality to get

$$\begin{aligned} \bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_i\varphi_i,\sum_{j\in\mathcal{J}}y_j\varphi_j \bigg\rangle\bigg| &=\bigg|\bigg\langle \sum_{i\in\mathcal{I}}\sum_{k_1=0}^3x_{i,k_1}\mathrm{i}^{k_1}\varphi_i,\sum_{j\in\mathcal{J}}\sum_{k_2=0}^3y_{j,k_2}\mathrm{i}^{k_2}\varphi_j \bigg\rangle\bigg|\\ &\leq\sum_{k_1=0}^3\sum_{k_2=0}^3\bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_{i,k_1}\varphi_i,\sum_{j\in\mathcal{J}}y_{j,k_2}\varphi_j \bigg\rangle\bigg|. \end{aligned}$$

Finally, we normalize the coefficients by \((\sum_{i\in\mathcal{I}}x_{i,k_{1}}^{2})^{1/2}\) and \((\sum_{j\in\mathcal{J}}y_{j,k_{2}}^{2})^{1/2}\) so we can apply our second bound:

$$\begin{aligned} \bigg|\bigg\langle \sum_{i\in\mathcal{I}}x_i\varphi_i,\sum_{j\in\mathcal{J}}y_j\varphi_j \bigg\rangle\bigg| &\leq\sum_{k_1=0}^3\sum_{k_2=0}^3\bigg(\sum_{i\in\mathcal{I}}x_{i,k_1}^2\bigg)^{1/2}\bigg(\sum_{j\in\mathcal{J}}y_{j,k_2}^2\bigg)^{1/2} C'\hat{\theta}\log K\\ &\leq(C\hat{\theta}\log K)\|x\|\|y\|, \end{aligned}$$

where C=4C′≈74.17 by the Cauchy-Schwarz inequality, and so we are done. □

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bandeira, A.S., Fickus, M., Mixon, D.G. et al. The Road to Deterministic Matrices with the Restricted Isometry Property. J Fourier Anal Appl 19, 1123–1149 (2013). https://doi.org/10.1007/s00041-013-9293-2

Download citation

Keywords

  • Restricted isometry property
  • Compressed sensing
  • Equiangular tight frames

Mathematics Subject Classification

  • 15A42
  • 05E30
  • 15B52
  • 60F10
  • 94A12