Skip to main content

Analysis of Sparse Recovery Algorithms via the Replica Method

  • 275 Accesses

Part of the Applied and Numerical Harmonic Analysis book series (ANHA)

Abstract

This chapter goes through the fundamental connections between statistical mechanics and estimation theory by focusing on the particular problem of compressive sensing. We first show that the asymptotic analysis of a sparse recovery algorithm is mathematically equivalent to the problem of calculating the free energy of a spin glass in the thermodynamic limit. We then use the replica method from statistical mechanics to evaluate the performance in the asymptotic regime. The asymptotic results have several applications in communications and signal processing. We briefly go through two instances of these applications: Characterization of joint sparse recovery algorithms used in distributed compressive sensing and tuning of receivers employed for detection of spatially modulated signals.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-09745-4_5
  • Chapter length: 35 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   119.00
Price excludes VAT (USA)
  • ISBN: 978-3-031-09745-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   159.99
Price excludes VAT (USA)
Fig. 5.1
Fig. 5.2

Notes

  1. 1.

    This is typically the case in classic signal sampling techniques.

  2. 2.

    Note that for random S, the density of state is random.

  3. 3.

    A Haar matrix is a random matrix generated from the rotation-invariant measure on the set of all orthonormal matrices.

  4. 4.

    Another well-known example in compressive sensing is the row-orthonormal random sensing matrix; see [39] and references therein for the exact definition and further examples.

  5. 5.

    By support, we refer to the indices of non-zero entries in a vector.

  6. 6.

    Note that the index j is dropped, as we consider a single-terminal setting.

  7. 7.

    Or to learning algorithms in a Bayesian framework.

  8. 8.

    Note that this is the case for any choice of the regularization function.

  9. 9.

    More precisely, if \(F_j \left ( \lambda \right )\) is different from a step function at a single mass point, i.e., derivative of \(F_j \left ( \lambda \right )\) is different from a Dirac impulse at a single mass point, \(\mbox{R}_j \left ( w \right )\) is strictly increasing; for details, see [59, Appendix E].

  10. 10.

    This follows the same reasons given in Sect. 5.4 for the asymptotic analysis of RLS-based algorithms.

  11. 11.

    In fact, the main two macroscopic parameters of a thermodynamic system are entropy and energy. The free energy is derived by applying the second law of thermodynamics as the Lagrange dual function. We however use directly the free energy in our formulation, for sake of brevity.

  12. 12.

    This means that there is no energy flow.

  13. 13.

    In statistical mechanics, this randomizer is known to have quenched randomness. This is different from the type of randomness considered for the microstate.

  14. 14.

    Here, the expectation is taken over the randomizer Ω.

  15. 15.

    Remember that for some choices of regularization function, e.g., 0-norm, this problem is not even numerically solvable.

  16. 16.

    Note that the expectation is taken with respect to all random variables.

  17. 17.

    In the notation, we drop the integration set for sake of compactness.

  18. 18.

    There are in general various ways to test the validity. The most common test is the zero-temperature entropy test; see [2]. For computationally feasible approaches, one can compare the given solution with large-dimensional (but still finite-dimensional) simulations; for instance, see the consistency the RS solution with numerical simulations in [2, Chapter 6] for 1-norm minimization.

  19. 19.

    The invalidity of the solution in these cases is shown by the zero-temperature entropy test. For some particular cases, the RS solution violates the known rigorous bounds.

  20. 20.

    For sake of brevity, we skip the detailed system model. Interested readers are referred to [11, 16] and the references therein.

  21. 21.

    Since temporal correlation is usually avoided by classic sampling approaches.

References

  1. Asaad, S., Bereyhi, A., Müller, R.R., Schaefer, R.F.: Joint user selection and precoding in multiuser MIMO systems via group LASSO. In: Proceedings of the IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Istanbul (2019)

    Google Scholar 

  2. Bereyhi, A.: Statistical Mechanics of Regularized Least Squares. Ph.D. Dissertation, Friedrich-Alexander University (2020)

    Google Scholar 

  3. Bereyhi, A., Müller, R.R.: Maximum-a-posteriori signal recovery with prior information: applications to compressive sensing. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, pp. 4494–4498 (2018)

    Google Scholar 

  4. Bereyhi, A., Müller, R., Schulz-Baldes, H.: RSB decoupling property of MAP estimators. In: Proceedings of the IEEE Information Theory Workshop (ITW),Cambridge, pp. 379–383 (2016)

    Google Scholar 

  5. Bereyhi, A., Müller, R.R., Schulz-Baldes, H.: Replica symmetry breaking in compressive sensing. In: Proceedings of the IEEE Information Theory and Applications Workshop (ITA), San Diego, pp. 1–7 (2017)

    Google Scholar 

  6. Bereyhi, A., Sedaghat, M.A., Asaad, S., Müller, R.: Nonlinear precoders for massive MIMO systems with general constraints. In: Proceedings of the VDE 21st International ITG Workshop on Smart Antennas (WSA), Berlin, pp. 1–8 (2017)

    Google Scholar 

  7. Bereyhi, A., Sedaghat, M.A., Müller, R.R.: Asymptotics of nonlinear LSE precoders with applications to transmit antenna selection. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), Aachen, pp. 81–85 (2017)

    Google Scholar 

  8. Bereyhi, A., Haghighatshoar, S., Müller, R.R.: Theoretical bounds on MAP estimation in distributed sensing networks. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), Vail (2018)

    Google Scholar 

  9. Bereyhi, A., Sedaghat, M.A., Müller, R.R.: RLS recovery with asymmetric penalty: fundamental limits and algorithmic approaches. In: Proceedings of the 2nd International Balkan Conference on Communications and Networking, Podgorica (2018)

    Google Scholar 

  10. Bereyhi, A., Sedaghat, M.A., Müller, R.R.: Precoding via approximate message passing with instantaneous signal constraints. In: Proceedings of the International Zurich Seminar on Information and Communication (IZS), Zürich, pp. 128–132 (2018)

    Google Scholar 

  11. Bereyhi, A., Asaad, S., Gäde, B., Müller, R.R.: RLS-based detection for massive spatial modulation MIMO. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), Paris, pp. 1167–1171 (2019)

    Google Scholar 

  12. Bereyhi, A., Asaad, S., Müller, R.R., Chatzinotas, S.: RLS precoding for massive MIMO systems with nonlinear front-end. In: Proceedings of the IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), Cannes (2019)

    Google Scholar 

  13. Bereyhi, A., Müller, R.R., Schulz-Baldes, H.: Statistical mechanics of MAP estimation: general replica ansatz. IEEE Trans. Inf. Theory 65(12), 7896–7934 (2019)

    CrossRef  MathSciNet  MATH  Google Scholar 

  14. Bereyhi, A., Jamali, V., Müller, R.R., Fischer, G., Schober, R., Tulino, A.M.: PAPR-limited precoding in massive MIMO systems with reflect- and transmit-array antennas. In: Proceedings of the Asilomar Conference on Signals, Systems, and Computers, Pacific Grove (2019). https://ieeexplore.ieee.org/abstract/document/9048889

  15. Bereyhi, A., Sedaghat, M.A., Müller, R.R., Fischer, G.: GLSE precoders for massive MIMO systems: analysis and applications. IEEE Trans. Wirel. Commun. 18(9), 4450–4465 (2019)

    CrossRef  Google Scholar 

  16. Bereyhi, A., Asaad, S., Gäde, B., Müller, R.R., Poor, H.V., “Detection of Spatially Modulated Signals via RLS: Theoretical Bounds and Applications,” in IEEE Transactions on Wireless Communications, vol. 21, no. 4, pp. 2291–2304, (2022). https://doi.org/10.1109/TWC.2021.3110839.

    CrossRef  Google Scholar 

  17. Chandrasekaran, V., Recht, B., Parrilo, P.A., Willsky, A.S.: The convex geometry of linear inverse problems. Found. Comput. Math. 12(6), 805–849 (2012)

    CrossRef  MathSciNet  MATH  Google Scholar 

  18. Davies, M.E., Eldar, Y.C.: Rank awareness in joint sparse recovery. IEEE Trans. Inf. Theory 58(2), 1135–1146 (2012)

    CrossRef  MathSciNet  MATH  Google Scholar 

  19. Donoho, D.L., Tanner, J.: Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. 102(27), 9452–9457 (2005)

    CrossRef  MathSciNet  MATH  Google Scholar 

  20. Donoho, D., Tanner, J.: Counting faces of randomly projected polytopes when the projection radically lowers dimension. J. Am. Math. Soc. 22(1), 1–53 (2009)

    CrossRef  MathSciNet  MATH  Google Scholar 

  21. Eldar, Y.C., Rauhut, H.: Average case analysis of multichannel sparse recovery using convex relaxation. IEEE Trans. Inf. Theory 56(1), 505–519 (2009)

    CrossRef  MathSciNet  MATH  Google Scholar 

  22. El Gamal, A., Kim, Y.H.: Network Information Theory. Cambridge University Press, Cambridge (2011)

    CrossRef  MATH  Google Scholar 

  23. Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Springer, New York (2013)

    CrossRef  MATH  Google Scholar 

  24. Gribonval, R., Rauhut, H., Schnass, K., Vandergheynst, P.: Atoms of all channels, unite! average case analysis of multi-channel sparse recovery using greedy algorithms. J. Fourier Anal. Appl. 14(5–6), 655–687 (2008)

    CrossRef  MathSciNet  MATH  Google Scholar 

  25. Guerra, F., Toninelli, F.L.: The thermodynamic limit in mean field spin glass models. Commun. Math. Phys. 230(1), 71–79 (2002)

    CrossRef  MathSciNet  MATH  Google Scholar 

  26. Guerra, F., Toninelli, F.L.: The infinite volume limit in generalized mean field disordered models. Markov Processes Relat. Fields 9, 195–207 (2003)

    MathSciNet  MATH  Google Scholar 

  27. Guionnet, A., Zeitouni, O.: Large deviations asymptotics for spherical integrals. J. Funct. Anal. 188(2), 461–515 (2002)

    CrossRef  MathSciNet  MATH  Google Scholar 

  28. Guo, D., Rasmussen, L.K., Lim, T.J.: Linear parallel interference cancellation in long-code CDMA multiuser detection. IEEE J. Sel. Areas Commun. 17(12), 2074–2081 (1999)

    CrossRef  Google Scholar 

  29. Guo, D., Verdú, S.: Randomly spread CDMA: asymptotics via statistical physics. IEEE Trans. Inf. Theory 51(6), 1983–2010 (2005)

    CrossRef  MathSciNet  MATH  Google Scholar 

  30. Guo, D., Baron, D., Shamai, S.: A single-letter characterization of optimal noisy compressed sensing. In: 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 52–59 (2009)

    Google Scholar 

  31. Haghighatshoar, S.: Multi terminal probabilistic compressed sensing. In: 2014 IEEE International Symposium on Information Theory, pp. 221–225 (2014)

    Google Scholar 

  32. Harish-Chandra: Differential operators on a semisimple Lie algebra. Am. J. Math., 87–120 (1957)

    Google Scholar 

  33. Itzykson, C., Zuber, J.B.: The planar approximation. II. J. Math. Phys. 21(3), 411–421 (1980)

    CrossRef  MathSciNet  MATH  Google Scholar 

  34. James, G.M., Paulson, C., Rusmevichientong, P.: The constrained lasso. Proc. Refereed Conf. Proc. 31, 4945–4950 (2012)

    Google Scholar 

  35. Kabashima, Y., Wadayama, T., Tanaka, T.: A typical reconstruction limit for compressed sensing based on p-norm minimization. J. Stat. Mech: Theory Exp. 2009(09), L09003 (2009)

    CrossRef  Google Scholar 

  36. Kabashima, Y., Wadayama, T., Tanaka, T.: Statistical mechanical analysis of a typical reconstruction limit of compressed sensing. In: Proceedings of IEEE International Symposium on Information Theory (ISIT), pp. 1533–1537 (2010)

    Google Scholar 

  37. Merhav, N.: Statistical physics and information theory. Found. Trends Commun. Inf. Theory 6(1–2), 1–212 (2010)

    CrossRef  MATH  Google Scholar 

  38. Müller, R.R., Gerstacker, W.H.: On the capacity loss due to separation of detection and decoding. IEEE Trans. Inf. Theory 50(8), 1769–1778 (2004)

    CrossRef  MathSciNet  MATH  Google Scholar 

  39. Müller, R.R., Alfano, G., Zaidel, B.M., de Miguel, R.: Applications of large random matrices in communications engineering. Preprint. arXiv:1310.5479 (2013)

    Google Scholar 

  40. Müller, R.R., Bereyhi, A., Mecklenbräuker, C.F.: Oversampled adaptive sensing with random projections: Analysis and algorithmic approaches. In: Proceedings of the IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), Louisville, pp. 336–341 (2018)

    Google Scholar 

  41. Oymak, S., Tropp, J.A.: Universality laws for randomized dimension reduction, with applications. Inf. Infer. J IMA 7(3), 337–446 (2018)

    MathSciNet  MATH  Google Scholar 

  42. Parisi, G.: A sequence of approximated solutions to the SK model for spin glasses. J. Phys. A Math. Gen. 13(4), L115 (1980)

    CrossRef  Google Scholar 

  43. Pastur, L., Shcherbina, M.: Absence of self-averaging of the order parameter in the Sherrington-Kirkpatrick model. J. Stat. Phys. 62(1–2), 1–19 (1991)

    CrossRef  MathSciNet  Google Scholar 

  44. Rangan, S., Fletcher, A.K., Goyal, V.: Asymptotic analysis of MAP estimation via the replica method and applications to compressed sensing. IEEE Trans. Inf. Theory 58(3), 1902–1923 (2012)

    CrossRef  MathSciNet  MATH  Google Scholar 

  45. Riesz, F.: Sur les valeurs moyennes des fonctions. J. Lond. Math. Society 1(2), 120–121 (1930)

    CrossRef  MathSciNet  MATH  Google Scholar 

  46. Schram, V., Bereyhi, A., Zaech, J.N., Müller, R.R., Gerstacker, W.H.: Approximate message passing for indoor THz channel estimation. In: Proceedings of the 3rd International Balkan Conference on Communications and Networking, Skopje (2019)

    Google Scholar 

  47. Sedaghat, M.A., Bereyhi, A., Müller, R.: A new class of nonlinear precoders for hardware efficient massive MIMO systems. In: Proceedings of the IEEE International Conference on Communications (ICC), Paris (2017)

    Google Scholar 

  48. Sedaghat, M.A., Bereyhi, A., Müller, R.R.: Least square error precoders for massive MIMO with signal constraints: fundamental limits. IEEE Trans. Wireless Commun. 17(1), 667–679 (2018)

    CrossRef  Google Scholar 

  49. Shamai, S., Verdú, S.: The impact of frequency-flat fading on the spectral efficiency of CDMA. IEEE Trans. Inf. Theory 47(4), 1302–1327 (2001)

    CrossRef  MathSciNet  MATH  Google Scholar 

  50. Stojnic, M.: Various thresholds for 1-optimization in compressed sensing. Preprint. arXiv:0907.3666 (2009)

    Google Scholar 

  51. Stojnic, M.: Recovery thresholds for 1 optimization in binary compressed sensing. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), pp. 1593–1597 (2010)

    Google Scholar 

  52. Tanaka, T.: A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors. IEEE Trans. Inf. Theory 48(11), 2888–2910 (2002)

    CrossRef  MathSciNet  MATH  Google Scholar 

  53. Tulino, A.M., Verdú, S., Verdu, S.: Random matrix theory and wireless communications. In: Foundations and TrendsTM in Communications and Information Theory, Now Publishers, Boston (2004)

    Google Scholar 

  54. Tulino, A.M., Caire, G., Verdu, S., Shamai, S.: Support recovery with sparsely sampled free random matrices. IEEE Trans. Inf. Theory 59(7), 4243–4271 (2013)

    CrossRef  Google Scholar 

  55. Vehkaperä, M., Kabashima, Y., Chatterjee, S.: Analysis of regularized LS reconstruction and random matrix ensembles in compressed sensing. IEEE Trans. Inf. Theory 62(4), 2100–2124 (2016)

    CrossRef  MathSciNet  MATH  Google Scholar 

  56. Wen, C.K., Zhang, J., Wong, K.K., Chen, J.C., Yuen, C.: On sparse vector recovery performance in structurally orthogonal matrices via lasso. IEEE Trans. Signal Proces. 64(17), 4519–4533 (2016)

    CrossRef  MathSciNet  MATH  Google Scholar 

  57. Wu, Y.: Shannon theory for compressed sensing. Ph.D. Dissertation, Princeton University (2011)

    Google Scholar 

  58. Wu, Y., Verdú, S.: Optimal phase transitions in compressed sensing. IEEE Trans. Inf. Theory 58(10), 6241–6263 (2012)

    CrossRef  MathSciNet  MATH  Google Scholar 

  59. Zaidel, B.M., Müller, R.R., Moustakas, A.L., de Miguel, R.: Vector precoding for Gaussian MIMO broadcast channels: impact of replica symmetry breaking. IEEE Trans. Inf. Theory 58(3), 1413–1440 (2012)

    CrossRef  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ali Bereyhi .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Bereyhi, A., Müller, R.R., Schulz-Baldes, H. (2022). Analysis of Sparse Recovery Algorithms via the Replica Method. In: Kutyniok, G., Rauhut, H., Kunsch, R.J. (eds) Compressed Sensing in Information Processing. Applied and Numerical Harmonic Analysis. Birkhäuser, Cham. https://doi.org/10.1007/978-3-031-09745-4_5

Download citation