Skip to main content
SpringerLink
Log in

Analysis of Sparse Recovery Algorithms via the Replica Method

  • Chapter
  • First Online:
Compressed Sensing in Information Processing

Part of the book series: Applied and Numerical Harmonic Analysis ((ANHA))

  • 599 Accesses

Abstract

This chapter goes through the fundamental connections between statistical mechanics and estimation theory by focusing on the particular problem of compressive sensing. We first show that the asymptotic analysis of a sparse recovery algorithm is mathematically equivalent to the problem of calculating the free energy of a spin glass in the thermodynamic limit. We then use the replica method from statistical mechanics to evaluate the performance in the asymptotic regime. The asymptotic results have several applications in communications and signal processing. We briefly go through two instances of these applications: Characterization of joint sparse recovery algorithms used in distributed compressive sensing and tuning of receivers employed for detection of spatially modulated signals.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    This is typically the case in classic signal sampling techniques.

  2. 2.

    Note that for random S, the density of state is random.

  3. 3.

    A Haar matrix is a random matrix generated from the rotation-invariant measure on the set of all orthonormal matrices.

  4. 4.

    Another well-known example in compressive sensing is the row-orthonormal random sensing matrix; see [39] and references therein for the exact definition and further examples.

  5. 5.

    By support, we refer to the indices of non-zero entries in a vector.

  6. 6.

    Note that the index j is dropped, as we consider a single-terminal setting.

  7. 7.

    Or to learning algorithms in a Bayesian framework.

  8. 8.

    Note that this is the case for any choice of the regularization function.

  9. 9.

    More precisely, if \(F_j \left ( \lambda \right )\) is different from a step function at a single mass point, i.e., derivative of \(F_j \left ( \lambda \right )\) is different from a Dirac impulse at a single mass point, \(\mbox{R}_j \left ( w \right )\) is strictly increasing; for details, see [59, Appendix E].

  10. 10.

    This follows the same reasons given in Sect. 5.4 for the asymptotic analysis of RLS-based algorithms.

  11. 11.

    In fact, the main two macroscopic parameters of a thermodynamic system are entropy and energy. The free energy is derived by applying the second law of thermodynamics as the Lagrange dual function. We however use directly the free energy in our formulation, for sake of brevity.

  12. 12.

    This means that there is no energy flow.

  13. 13.

    In statistical mechanics, this randomizer is known to have quenched randomness. This is different from the type of randomness considered for the microstate.

  14. 14.

    Here, the expectation is taken over the randomizer Ω.

  15. 15.

    Remember that for some choices of regularization function, e.g., ℓ 0-norm, this problem is not even numerically solvable.

  16. 16.

    Note that the expectation is taken with respect to all random variables.

  17. 17.

    In the notation, we drop the integration set for sake of compactness.

  18. 18.

    There are in general various ways to test the validity. The most common test is the zero-temperature entropy test; see [2]. For computationally feasible approaches, one can compare the given solution with large-dimensional (but still finite-dimensional) simulations; for instance, see the consistency the RS solution with numerical simulations in [2, Chapter 6] for ℓ 1-norm minimization.

  19. 19.

    The invalidity of the solution in these cases is shown by the zero-temperature entropy test. For some particular cases, the RS solution violates the known rigorous bounds.

  20. 20.

    For sake of brevity, we skip the detailed system model. Interested readers are referred to [11, 16] and the references therein.

  21. 21.

    Since temporal correlation is usually avoided by classic sampling approaches.

References

  1. Asaad, S., Bereyhi, A., Müller, R.R., Schaefer, R.F.: Joint user selection and precoding in multiuser MIMO systems via group LASSO. In: Proceedings of the IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), Istanbul (2019)

    Google Scholar 

  2. Bereyhi, A.: Statistical Mechanics of Regularized Least Squares. Ph.D. Dissertation, Friedrich-Alexander University (2020)

    Google Scholar 

  3. Bereyhi, A., Müller, R.R.: Maximum-a-posteriori signal recovery with prior information: applications to compressive sensing. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, pp. 4494–4498 (2018)

    Google Scholar 

  4. Bereyhi, A., Müller, R., Schulz-Baldes, H.: RSB decoupling property of MAP estimators. In: Proceedings of the IEEE Information Theory Workshop (ITW),Cambridge, pp. 379–383 (2016)

    Google Scholar 

  5. Bereyhi, A., Müller, R.R., Schulz-Baldes, H.: Replica symmetry breaking in compressive sensing. In: Proceedings of the IEEE Information Theory and Applications Workshop (ITA), San Diego, pp. 1–7 (2017)

    Google Scholar 

  6. Bereyhi, A., Sedaghat, M.A., Asaad, S., Müller, R.: Nonlinear precoders for massive MIMO systems with general constraints. In: Proceedings of the VDE 21st International ITG Workshop on Smart Antennas (WSA), Berlin, pp. 1–8 (2017)

    Google Scholar 

  7. Bereyhi, A., Sedaghat, M.A., Müller, R.R.: Asymptotics of nonlinear LSE precoders with applications to transmit antenna selection. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), Aachen, pp. 81–85 (2017)

    Google Scholar 

  8. Bereyhi, A., Haghighatshoar, S., Müller, R.R.: Theoretical bounds on MAP estimation in distributed sensing networks. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), Vail (2018)

    Google Scholar 

  9. Bereyhi, A., Sedaghat, M.A., Müller, R.R.: RLS recovery with asymmetric penalty: fundamental limits and algorithmic approaches. In: Proceedings of the 2nd International Balkan Conference on Communications and Networking, Podgorica (2018)

    Google Scholar 

  10. Bereyhi, A., Sedaghat, M.A., Müller, R.R.: Precoding via approximate message passing with instantaneous signal constraints. In: Proceedings of the International Zurich Seminar on Information and Communication (IZS), Zürich, pp. 128–132 (2018)

    Google Scholar 

  11. Bereyhi, A., Asaad, S., Gäde, B., Müller, R.R.: RLS-based detection for massive spatial modulation MIMO. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), Paris, pp. 1167–1171 (2019)

    Google Scholar 

  12. Bereyhi, A., Asaad, S., Müller, R.R., Chatzinotas, S.: RLS precoding for massive MIMO systems with nonlinear front-end. In: Proceedings of the IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), Cannes (2019)

    Google Scholar 

  13. Bereyhi, A., Müller, R.R., Schulz-Baldes, H.: Statistical mechanics of MAP estimation: general replica ansatz. IEEE Trans. Inf. Theory 65(12), 7896–7934 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  14. Bereyhi, A., Jamali, V., Müller, R.R., Fischer, G., Schober, R., Tulino, A.M.: PAPR-limited precoding in massive MIMO systems with reflect- and transmit-array antennas. In: Proceedings of the Asilomar Conference on Signals, Systems, and Computers, Pacific Grove (2019). https://ieeexplore.ieee.org/abstract/document/9048889

  15. Bereyhi, A., Sedaghat, M.A., Müller, R.R., Fischer, G.: GLSE precoders for massive MIMO systems: analysis and applications. IEEE Trans. Wirel. Commun. 18(9), 4450–4465 (2019)

    Article  Google Scholar 

  16. Bereyhi, A., Asaad, S., Gäde, B., Müller, R.R., Poor, H.V., “Detection of Spatially Modulated Signals via RLS: Theoretical Bounds and Applications,” in IEEE Transactions on Wireless Communications, vol. 21, no. 4, pp. 2291–2304, (2022). https://doi.org/10.1109/TWC.2021.3110839.

    Article  Google Scholar 

  17. Chandrasekaran, V., Recht, B., Parrilo, P.A., Willsky, A.S.: The convex geometry of linear inverse problems. Found. Comput. Math. 12(6), 805–849 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  18. Davies, M.E., Eldar, Y.C.: Rank awareness in joint sparse recovery. IEEE Trans. Inf. Theory 58(2), 1135–1146 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  19. Donoho, D.L., Tanner, J.: Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. 102(27), 9452–9457 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  20. Donoho, D., Tanner, J.: Counting faces of randomly projected polytopes when the projection radically lowers dimension. J. Am. Math. Soc. 22(1), 1–53 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  21. Eldar, Y.C., Rauhut, H.: Average case analysis of multichannel sparse recovery using convex relaxation. IEEE Trans. Inf. Theory 56(1), 505–519 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  22. El Gamal, A., Kim, Y.H.: Network Information Theory. Cambridge University Press, Cambridge (2011)

    Book  MATH  Google Scholar 

  23. Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Springer, New York (2013)

    Book  MATH  Google Scholar 

  24. Gribonval, R., Rauhut, H., Schnass, K., Vandergheynst, P.: Atoms of all channels, unite! average case analysis of multi-channel sparse recovery using greedy algorithms. J. Fourier Anal. Appl. 14(5–6), 655–687 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  25. Guerra, F., Toninelli, F.L.: The thermodynamic limit in mean field spin glass models. Commun. Math. Phys. 230(1), 71–79 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  26. Guerra, F., Toninelli, F.L.: The infinite volume limit in generalized mean field disordered models. Markov Processes Relat. Fields 9, 195–207 (2003)

    MathSciNet  MATH  Google Scholar 

  27. Guionnet, A., Zeitouni, O.: Large deviations asymptotics for spherical integrals. J. Funct. Anal. 188(2), 461–515 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  28. Guo, D., Rasmussen, L.K., Lim, T.J.: Linear parallel interference cancellation in long-code CDMA multiuser detection. IEEE J. Sel. Areas Commun. 17(12), 2074–2081 (1999)

    Article  Google Scholar 

  29. Guo, D., Verdú, S.: Randomly spread CDMA: asymptotics via statistical physics. IEEE Trans. Inf. Theory 51(6), 1983–2010 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  30. Guo, D., Baron, D., Shamai, S.: A single-letter characterization of optimal noisy compressed sensing. In: 47th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 52–59 (2009)

    Google Scholar 

  31. Haghighatshoar, S.: Multi terminal probabilistic compressed sensing. In: 2014 IEEE International Symposium on Information Theory, pp. 221–225 (2014)

    Google Scholar 

  32. Harish-Chandra: Differential operators on a semisimple Lie algebra. Am. J. Math., 87–120 (1957)

    Google Scholar 

  33. Itzykson, C., Zuber, J.B.: The planar approximation. II. J. Math. Phys. 21(3), 411–421 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  34. James, G.M., Paulson, C., Rusmevichientong, P.: The constrained lasso. Proc. Refereed Conf. Proc. 31, 4945–4950 (2012)

    Google Scholar 

  35. Kabashima, Y., Wadayama, T., Tanaka, T.: A typical reconstruction limit for compressed sensing based on ℓ p-norm minimization. J. Stat. Mech: Theory Exp. 2009(09), L09003 (2009)

    Article  Google Scholar 

  36. Kabashima, Y., Wadayama, T., Tanaka, T.: Statistical mechanical analysis of a typical reconstruction limit of compressed sensing. In: Proceedings of IEEE International Symposium on Information Theory (ISIT), pp. 1533–1537 (2010)

    Google Scholar 

  37. Merhav, N.: Statistical physics and information theory. Found. Trends Commun. Inf. Theory 6(1–2), 1–212 (2010)

    Article  MATH  Google Scholar 

  38. Müller, R.R., Gerstacker, W.H.: On the capacity loss due to separation of detection and decoding. IEEE Trans. Inf. Theory 50(8), 1769–1778 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  39. Müller, R.R., Alfano, G., Zaidel, B.M., de Miguel, R.: Applications of large random matrices in communications engineering. Preprint. arXiv:1310.5479 (2013)

    Google Scholar 

  40. Müller, R.R., Bereyhi, A., Mecklenbräuker, C.F.: Oversampled adaptive sensing with random projections: Analysis and algorithmic approaches. In: Proceedings of the IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), Louisville, pp. 336–341 (2018)

    Google Scholar 

  41. Oymak, S., Tropp, J.A.: Universality laws for randomized dimension reduction, with applications. Inf. Infer. J IMA 7(3), 337–446 (2018)

    MathSciNet  MATH  Google Scholar 

  42. Parisi, G.: A sequence of approximated solutions to the SK model for spin glasses. J. Phys. A Math. Gen. 13(4), L115 (1980)

    Article  Google Scholar 

  43. Pastur, L., Shcherbina, M.: Absence of self-averaging of the order parameter in the Sherrington-Kirkpatrick model. J. Stat. Phys. 62(1–2), 1–19 (1991)

    Article  MathSciNet  Google Scholar 

  44. Rangan, S., Fletcher, A.K., Goyal, V.: Asymptotic analysis of MAP estimation via the replica method and applications to compressed sensing. IEEE Trans. Inf. Theory 58(3), 1902–1923 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  45. Riesz, F.: Sur les valeurs moyennes des fonctions. J. Lond. Math. Society 1(2), 120–121 (1930)

    Article  MathSciNet  MATH  Google Scholar 

  46. Schram, V., Bereyhi, A., Zaech, J.N., Müller, R.R., Gerstacker, W.H.: Approximate message passing for indoor THz channel estimation. In: Proceedings of the 3rd International Balkan Conference on Communications and Networking, Skopje (2019)

    Google Scholar 

  47. Sedaghat, M.A., Bereyhi, A., Müller, R.: A new class of nonlinear precoders for hardware efficient massive MIMO systems. In: Proceedings of the IEEE International Conference on Communications (ICC), Paris (2017)

    Google Scholar 

  48. Sedaghat, M.A., Bereyhi, A., Müller, R.R.: Least square error precoders for massive MIMO with signal constraints: fundamental limits. IEEE Trans. Wireless Commun. 17(1), 667–679 (2018)

    Article  Google Scholar 

  49. Shamai, S., Verdú, S.: The impact of frequency-flat fading on the spectral efficiency of CDMA. IEEE Trans. Inf. Theory 47(4), 1302–1327 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  50. Stojnic, M.: Various thresholds for ℓ 1-optimization in compressed sensing. Preprint. arXiv:0907.3666 (2009)

    Google Scholar 

  51. Stojnic, M.: Recovery thresholds for ℓ 1 optimization in binary compressed sensing. In: Proceedings of the IEEE International Symposium on Information Theory (ISIT), pp. 1593–1597 (2010)

    Google Scholar 

  52. Tanaka, T.: A statistical-mechanics approach to large-system analysis of CDMA multiuser detectors. IEEE Trans. Inf. Theory 48(11), 2888–2910 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  53. Tulino, A.M., Verdú, S., Verdu, S.: Random matrix theory and wireless communications. In: Foundations and TrendsTM in Communications and Information Theory, Now Publishers, Boston (2004)

    Google Scholar 

  54. Tulino, A.M., Caire, G., Verdu, S., Shamai, S.: Support recovery with sparsely sampled free random matrices. IEEE Trans. Inf. Theory 59(7), 4243–4271 (2013)

    Article  Google Scholar 

  55. Vehkaperä, M., Kabashima, Y., Chatterjee, S.: Analysis of regularized LS reconstruction and random matrix ensembles in compressed sensing. IEEE Trans. Inf. Theory 62(4), 2100–2124 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  56. Wen, C.K., Zhang, J., Wong, K.K., Chen, J.C., Yuen, C.: On sparse vector recovery performance in structurally orthogonal matrices via lasso. IEEE Trans. Signal Proces. 64(17), 4519–4533 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  57. Wu, Y.: Shannon theory for compressed sensing. Ph.D. Dissertation, Princeton University (2011)

    Google Scholar 

  58. Wu, Y., Verdú, S.: Optimal phase transitions in compressed sensing. IEEE Trans. Inf. Theory 58(10), 6241–6263 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  59. Zaidel, B.M., Müller, R.R., Moustakas, A.L., de Miguel, R.: Vector precoding for Gaussian MIMO broadcast channels: impact of replica symmetry breaking. IEEE Trans. Inf. Theory 58(3), 1413–1440 (2012)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Digital Communications, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany

    Ali Bereyhi & Ralf R. Müller

  2. Department of Mathematics, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany

    Hermann Schulz-Baldes

Authors
  1. Ali Bereyhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ralf R. Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hermann Schulz-Baldes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ali Bereyhi .

Editor information

Editors and Affiliations

  1. Mathematisches Institut, Ludwig Maximilian University of Munich, München, Bayern, Germany

    Gitta Kutyniok

  2. Lehrstuhl für Mathematik, RWTH Aachen University, Aachen, Nordrhein-Westfalen, Germany

    Holger Rauhut

  3. Lehrstuhl für Mathematik, RWTH Aachen University, Aachen, Nordrhein-Westfalen, Germany

    Robert J. Kunsch

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Bereyhi, A., Müller, R.R., Schulz-Baldes, H. (2022). Analysis of Sparse Recovery Algorithms via the Replica Method. In: Kutyniok, G., Rauhut, H., Kunsch, R.J. (eds) Compressed Sensing in Information Processing. Applied and Numerical Harmonic Analysis. Birkhäuser, Cham. https://doi.org/10.1007/978-3-031-09745-4_5

Download citation

Publish with us

Policies and ethics

Search

Navigation