Recovering Structured Signals in Noise: Least-Squares Meets Compressed Sensing

  • Christos Thrampoulidis
  • Samet Oymak
  • Babak Hassibi
Part of the Applied and Numerical Harmonic Analysis book series (ANHA)


The typical scenario that arises in most “big data” problems is one where the ambient dimension of the signal is very large (e.g., high resolution images, gene expression data from a DNA microarray, social network data, etc.), yet is such that its desired properties lie in some low dimensional structure (sparsity, low-rankness, clusters, etc.). In the modern viewpoint, the goal is to come up with efficient algorithms to reveal these structures and for which, under suitable conditions, one can give theoretical guarantees. We specifically consider the problem of recovering such a structured signal (sparse, low-rank, block-sparse, etc.) from noisy compressed measurements. A general algorithm for such problems, commonly referred to as generalized LASSO, attempts to solve this problem by minimizing a least-squares cost with an added “structure-inducing” regularization term (1 norm, nuclear norm, mixed 2/1 norm, etc.). While the LASSO algorithm has been around for 20 years and has enjoyed great success in practice, there has been relatively little analysis of its performance. In this chapter, we will provide a full performance analysis and compute, in closed form, the mean-square-error of the reconstructed signal. We will highlight some of the mathematical vignettes necessary for the analysis, make connections to noiseless compressed sensing and proximal denoising, and will emphasize the central role of the “statistical dimension” of a structured signal.



The authors gratefully acknowledge the anonymous reviewers for their attention and their helpful comments.


  1. 1.
    Amelunxen, D., Lotz, M., McCoy, M.B., Tropp, J.A.: Living on the edge: a geometric theory of phase transitions in convex optimization. arXiv preprint. arXiv:1303.6672 (2013)Google Scholar
  2. 2.
    Bayati, M., Montanari, A.: The dynamics of message passing on dense graphs, with applications to compressed sensing. IEEE Trans. Inf. Theory 57(2), 764–785 (2011)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Bayati, M., Montanari, A.: The LASSO risk for gaussian matrices. IEEE Trans. Inf. Theory 58(4), 1997–2017 (2012)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Belloni, A., Chernozhukov, V., Wang, L.: Square-root lasso: pivotal recovery of sparse signals via conic programming. Biometrika 98(4), 791–806 (2011)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Bertsekas, D., Nedic, A., Ozdaglar, A.: Convex Analysis and Optimization. Athena Scientific (2003)Google Scholar
  6. 6.
    Bickel, P.J., Ritov, Y., Tsybakov, A.B.: Simultaneous analysis of Lasso and Dantzig selector. Ann. Stat. 37(4), 1705–1732 (2009)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Borwein, J.M., Lewis, A.S.: Convex Analysis and Nonlinear Optimization: Theory and Examples, vol. 3. Springer, New York (2010)Google Scholar
  8. 8.
    Cai, J.-F., Xu, W.: Guarantees of total variation minimization for signal recovery. arXiv preprint. arXiv:1301.6791 (2013)Google Scholar
  9. 9.
    Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9(6), 717–772 (2009)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Candes, E., Tao, T.: Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE Trans. Inf. Theory 52(12),5406–5425 (2006)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Candes, E., Tao, T.: The Dantzig selector: statistical estimation when p is much larger than n. Ann. Stat. 35, 2313–2351 (2007)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Candes, E.J., Romberg, J.K., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Chandrasekaran, V., Jordan, M.I.: Computational and statistical tradeoffs via convex relaxation. Proc. Natl. Acad. Sci. 110(13), E1181–E1190 (2013)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Chandrasekaran, V., Recht, B., Parrilo, P.A., Willsky, A.S.: The convex geometry of linear inverse problems. Found. Comput. Math. 12(6), 805–849 (2012)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(3), 613–627 (1995)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Donoho, D.L.: High-dimensional data analysis: the curses and blessings of dimensionality. Aide-memoire of a lecture at “AMS Conference on Math Challenges of the 21st Century”. Citeseer (2000)Google Scholar
  17. 17.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Donoho, D.L.: High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension. Discrete Comput. Geom. 35(4), 617–652 (2006)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Donoho, D.L., Tanner, J.: Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. USA 102(27), 9452–9457 (2005)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Donoho, D.L., Tanner, J.: Sparse nonnegative solution of underdetermined linear equations by linear programming. Proc. Natl. Acad. Sci. USA 102(27), 9446–9451 (2005)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Donoho, D.L., Tanner, J.: Thresholds for the recovery of sparse solutions via l1 minimization. In: The 40th Annual Conference onInformation Sciences and Systems, 2006, pp. 202–206. IEEE, New York (2006)Google Scholar
  22. 22.
    Donoho, D., Tanner, J.: Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Philos. Trans. Roy. Soc. A Math. Phys. Eng. Sci. 367(1906), 4273–4293 (2009)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Donoho, D.L., Tanner, J.: Precise undersampling theorems. Proc. IEEE 98(6), 913–924 (2010)CrossRefGoogle Scholar
  24. 24.
    Donoho, D.L., Elad, M., Temlyakov, V.N.: Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inf. Theory 52(1), 6–18 (2006)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Donoho, D.L., Maleki, A., Montanari, A.: Message-passing algorithms for compressed sensing. Proc. Natl. Acad. Sci. 106(45), 18914–18919 (2009)CrossRefGoogle Scholar
  26. 26.
    Donoho, D.L., Maleki, A., Montanari, A.: The noise-sensitivity phase transition in compressed sensing. IEEE Trans. Inf. Theory 57(10), 6920–6941 (2011)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Donoho, D., Johnstone, I., Montanari, A.: Accurate prediction of phase transitions in compressed sensing via a connection to minimax denoising. IEEE Trans. Inf. Theory 59(6), 3396–3433 (2013)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Donoho, D.L., Gavish, M., Montanari, A.: The phase transition of matrix recovery from Gaussian measurements matches the minimax mse of matrix denoising. Proc. Natl. Acad. Sci. 110(21), 8405–8410 (2013)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Eldar, Y.C., Kuppinger, P., Bolcskei, H.: Block-sparse signals: uncertainty relations and efficient recovery. IEEE Trans. Signal Process. 58(6), 3042–3054 (2010)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Fazel, M.: Matrix rank minimization with applications. Ph.D. thesis (2002)Google Scholar
  31. 31.
    Foygel, R., Mackey, L.: Corrupted sensing: novel guarantees for separating structured signals. arXiv preprint. arXiv:1305.2524 (2013)Google Scholar
  32. 32.
    Gandy, S., Recht, B., Yamada, I.: Tensor completion and low-n-rank tensor recovery via convex optimization. Inverse Prob. 27(2), 025010 (2011)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Gordon, Y.: Some inequalities for Gaussian processes and applications. Isr. J. Math. 50(4), 265–289 (1985)CrossRefGoogle Scholar
  34. 34.
    Gordon, Y.: On Milman’s Inequality and Random Subspaces Which Escape Through a Mesh in \(\mathbb{R}^{n}\). Springer, New York (1988)Google Scholar
  35. 35.
    Härdle, W., Simar, L.: Applied Multivariate Statistical Analysis, vol. 2. Springer, Berlin (2007)Google Scholar
  36. 36.
    Kressner, D., Steinlechner, M., Vandereycken, B.: Low-rank tensor completion by Riemannian optimization. BIT Numer. Math. 54(2):447–468 (2014)MathSciNetCrossRefGoogle Scholar
  37. 37.
    Ledoux, M., Talagrand, M.: Probability in Banach Saces: Isoperimetry and Processes, vol. 23. Springer, Berlin (1991)CrossRefGoogle Scholar
  38. 38.
    Maleki, M.A.: Approximate Message Passing Algorithms for Compressed Sensing. Stanford University, Stanford (2010)Google Scholar
  39. 39.
    Maleki, A., Anitori, L., Yang, Z., Baraniuk, R.G.: Asymptotic analysis of complex lasso via complex approximate message passing (camp). IEEE Trans. Inf. Theory 59(7):4290–4308 (2013)MathSciNetCrossRefGoogle Scholar
  40. 40.
    McCoy, M.B., Tropp, J.A.: From Steiner formulas for cones to concentration of intrinsic volumes. Discrete Comput. Geom. 51(4), 926–963 (2014)MathSciNetCrossRefGoogle Scholar
  41. 41.
    Merriman, M.: On the history of the method of least squares. Analyst 4, 33–36 (1877)CrossRefGoogle Scholar
  42. 42.
    Negahban, S.N., Ravikumar, P., Wainwright, M.J., Yu, B.: A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers. Stat. Sci. 27(4), 538–557 (2012)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Oymak, S., Hassibi, B.: Sharp MSE bounds for proximal denoising. arXiv preprint. arXiv:1305.2714 (2013)Google Scholar
  44. 44.
    Oymak, S., Mohan, K., Fazel, M., Hassibi, B.: A simplified approach to recovery conditions for low rank matrices. In: IEEE International Symposium on Information Theory Proceedings (ISIT), 2011, pp. 2318–2322. IEEE, New York (2011)Google Scholar
  45. 45.
    Oymak, S., Jalali, A., Fazel, M., Eldar, Y.C., Hassibi, B.: Simultaneously structured models with application to sparse and low-rank matrices. arXiv preprint. arXiv:1212.3753 (2012)Google Scholar
  46. 46.
    Oymak, S., Thrampoulidis, C., Hassibi, B.: Simple bounds for noisy linear inverse problems with exact side information. arXiv preprint. arXiv:1312.0641 (2013)Google Scholar
  47. 47.
    Oymak, S., Thrampoulidis, C., Hassibi, B.: The squared-error of generalized LASSO: a precise analysis. arXiv preprint. arXiv:1311.0830 (2013)Google Scholar
  48. 48.
    Rao, N., Recht, B., Nowak, R.: Tight measurement bounds for exact recovery of structured sparse signals. arXiv preprint. arXiv:1106.4355 (2011)Google Scholar
  49. 49.
    Raskutti, G., Wainwright, M.J., Yu, B.: Restricted eigenvalue properties for correlated Gaussian designs. J. Mach. Learn. Res. 99, 2241–2259 (2010)MathSciNetGoogle Scholar
  50. 50.
    Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Richard, E., Savalle, P.-A., Vayatis, N.: Estimation of simultaneously sparse and low rank matrices. arXiv preprint. arXiv:1206.6474 (2012)Google Scholar
  52. 52.
    Rockafellar, R.T.: Convex Analysis, vol. 28. Princeton University Press, Princeton (1997)Google Scholar
  53. 53.
    Stigler, S.M.: Gauss and the invention of least squares. Ann. Stat. 9, 465–474 (1981)MathSciNetCrossRefGoogle Scholar
  54. 54.
    Stojnic, M.: Block-length dependent thresholds in block-sparse compressed sensing. arXiv preprint. arXiv:0907.3679 (2009)Google Scholar
  55. 55.
    Stojnic, M.: Various thresholds for 1-optimization in compressed sensing. arXiv preprint. arXiv:0907.3666 (2009)Google Scholar
  56. 56.
    Stojnic, M.: A framework to characterize performance of LASSO algorithms. arXiv preprint. arXiv:1303.7291 (2013)Google Scholar
  57. 57.
    Stojnic, M.: A rigorous geometry-probability equivalence in characterization of 1-optimization. arXiv preprint. arXiv:1303.7287 (2013)Google Scholar
  58. 58.
    Stojnic, M., Parvaresh, F., Hassibi, B.: On the reconstruction of block-sparse signals with an optimal number of measurements. IEEE Trans. Signal Process. 57(8), 3075–3085 (2009)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Taylor, J., et al.: The geometry of least squares in the 21st century. Bernoulli 19(4), 1449–1464 (2013)Google Scholar
  60. 60.
    Thrampoulidis, C., Oymak, S., Hassibi, B.: Simple error bounds for regularized noisy linear inverse problems. In: 2014 IEEE International Symposium on Information Theory (ISIT), pp. 3007–3011. IEEE (2014)Google Scholar
  61. 61.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Roy. Stat. Soc. Ser. B (Methodological) 58, 267–288 (1996)MathSciNetGoogle Scholar
  62. 62.
  63. 63.
    Vershynin, R.: Introduction to the non-asymptotic analysis of random matrices. arXiv preprint. arXiv:1011.3027 (2010)Google Scholar
  64. 64.
    Wainwright, M.J.: Sharp thresholds for high-dimensional and noisy sparsity recovery using-constrained quadratic programming (Lasso). IEEE Trans. Inf. Theory 55(5), 2183–2202 (2009)MathSciNetCrossRefGoogle Scholar
  65. 65.
    Wright, J., Ganesh, A., Min, K., Ma, Y.: Compressive principal component pursuit. Inf. Infer. 2(1), 32–68 (2013)MathSciNetCrossRefGoogle Scholar
  66. 66.
    Zhao, P., Yu, B.: On model selection consistency of Lasso. J. Mach. Learn. Res. 7, 2541–2563 (2006)MathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Christos Thrampoulidis
    • 1
  • Samet Oymak
    • 1
  • Babak Hassibi
    • 1
  1. 1.California Institute of TechnologyPasadenaUSA

Personalised recommendations