Matrix-free interior point method for compressed sensing problems

Abstract

We consider a class of optimization problems for sparse signal reconstruction which arise in the field of compressed sensing (CS). A plethora of approaches and solvers exist for such problems, for example GPSR, FPC_AS, SPGL1, NestA, \(\mathbf{\ell _1\_\ell _s}\), PDCO to mention a few. CS applications lead to very well conditioned optimization problems and therefore can be solved easily by simple first-order methods. Interior point methods (IPMs) rely on the Newton method hence they use the second-order information. They have numerous advantageous features and one clear drawback: being the second-order approach they need to solve linear equations and this operation has (in the general dense case) an \({\mathcal {O}}(n^3)\) computational complexity. Attempts have been made to specialize IPMs to sparse reconstruction problems and they have led to interesting developments implemented in \(\mathbf{\ell _1\_\ell _s}\) and PDCO softwares. We go a few steps further. First, we use the matrix-free IPM, an approach which redesigns IPM to avoid the need to explicitly formulate (and store) the Newton equation systems. Secondly, we exploit the special features of the signal processing matrices within the matrix-free IPM. Two such features are of particular interest: an excellent conditioning of these matrices and the ability to perform inexpensive (low complexity) matrix–vector multiplications with them. Computational experience with large scale one-dimensional signals confirms that the new approach is efficient and offers an attractive alternative to other state-of-the-art solvers.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. 1.

    Ahmed, N., Rao, K.R.: Orthogonal transforms for digital signal processing. Springer, Berlin (1975)

    Book  MATH  Google Scholar 

  2. 2.

    Baraniuk, R., Davenport, M., DeVore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28(3), 253–263 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  3. 3.

    Becker, S., Bobin, J., Candés, E.J.: Nesta: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4(1), 1–39 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  4. 4.

    Van Den Berg, E., Friedlander, M.P.: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31(2), 890–912 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  5. 5.

    Van Den Berg, E., Friedlander, M.P., Hennenfent, G., Herrman, F.J., Saab, R., Yılmaz, Ö.: Sparco: a testing framework for sparse reconstruction. ACM Trans. Math. Softw. 35(4), 1–16 (2009)

    Article  Google Scholar 

  6. 6.

    Blanchard, J.D., Cartis, C., Tanner, J.: Decay properties of restricted isometry constants. Comput. Optim. Appl. 16(7), 572–575 (2009)

    Google Scholar 

  7. 7.

    Blanchard, J.D., Cartis, C., Tanner, J.: Compressed sensing: how sharp is the restricted isometry property? SIAM Rev. 53(1), 105–125 (2011)

    Google Scholar 

  8. 8.

    Candés, E., Demanet, L., Donoho, D., Ying, L.: Curvelab. http://www.curvelet.org/software.html

  9. 9.

    Candés, E.J.: Compressive sampling. In: Proceedings of the International Congress of Mathematicians (2006)

  10. 10.

    Candés, E.J., Romberg, J.: Practical signal recovery from random projections. In: Proceedings of the SPIE Conference on Wavelet Applications in Signal and Image Processing XI, 5914 (2004)

  11. 11.

    Candés, E.J., Romberg, J.: \(\ell _1\)-magic. Technical Report, Caltech. http://users.ece.gatech.edu/justin/l1magic/ (2007)

  12. 12.

    Candés, E.J., Romberg, J.: Sparsity and incoherence in compressive sampling. Inverse Probl. 23(3), 969–985 (2007)

    Article  MATH  Google Scholar 

  13. 13.

    Candés, E.J., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Comm. Pure Appl. Math. 59(8), 1207–1223 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  14. 14.

    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    Article  MathSciNet  Google Scholar 

  15. 15.

    Coifman, R., Geshwind, F., Meyer, Y.: Noiselets. Appl. Comput. Harmon. Anal. 10(1), 27–44 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  16. 16.

    Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory 47(7), 2845–2862 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  17. 17.

    Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994)

    Article  MATH  MathSciNet  Google Scholar 

  18. 18.

    Donoho, D.L., Tanner, J.: Precise undersampling theorems. Proc. IEEE 98(6), 913–924 (2010)

    Article  Google Scholar 

  19. 19.

    Dossal, C., Mallat, S.: Sparse spike deconvolution with minimum scale. Proc. Signal Process. Adapt. Sparse Struct. Represent. 81(3), 123–126 (1994)

    Google Scholar 

  20. 20.

    Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1(4), 586–597 (2007)

    Article  Google Scholar 

  21. 21.

    Foucart, S.: A note on guaranteed sparse recovery via 1-minimization. Appl. Comput. Harmon. Anal. 29(1), 97–103 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  22. 22.

    Frigo, M., Johnston, S.G.: The design and implementation of FFTW3. Proc. IEEE 93(2), 216–231 (2005)

    Article  Google Scholar 

  23. 23.

    Gondzio, J.: Multiple centrality corrections in a primal–dual method for linear programming. Comput. Optim. Appl. 6, 137–156 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  24. 24.

    Gondzio, J.: Interior point methods 25 years later. Eur. J. Oper. Res. 218(3), 587–601 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  25. 25.

    Gondzio, J.: Matrix-free interior point method. Comput. Optim. Appl. 51(2), 457–480 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  26. 26.

    Gondzio, J.: Convergence analysis of an inexact feasible interior point method for convex quadratic programming. SIAM J. Optim. 23(3), 1510–1527 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  27. 27.

    Hennenfent, G., Herrmann, F.J.: Random sampling: new insights into the reconstruction of coarsely-sampled wavefields. In: SEG International Exposition and 77th Annual Meeting (2007)

  28. 28.

    Kelley, C.T.: Iterative methods for linear and nonlinear equations, volume 16 of Frontiers in Applied Mathematics. SIAM, Philadelphia (1995)

  29. 29.

    Kim, S.-J., Koh, K., Lustig, M., Boyd, S., Gorinevsky, D.: An interior-point method for large-scale \(\ell _1\)-regularized least squares. IEEE J. Sel. Top. Signal Process. 1(4), 606–617 (2007)

    Article  Google Scholar 

  30. 30.

    Kojima, M., Megiddo, N., Mizuno, S.: A primal-dual infeasible-interior-point algorithm for linear programming. Math. Progr. 61, 263–280 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  31. 31.

    Lu, Z., Monteiro, R.D.S., O’Neal, J.W.: An iterative solver-based infeasible primal–dual path-following algorithm for convex quadratic programming. SIAM J. Optim. 17, 287–310 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  32. 32.

    Miller, A.J.: Subset selection in regression. Chapmain & Hall/CRC, London (2002)

    Book  MATH  Google Scholar 

  33. 33.

    Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  34. 34.

    Rockafellar, R.T.: Convex analysis, Princeton landmarks in mathematics and physics. Princeton University Press, USA (1970)

    Google Scholar 

  35. 35.

    Rudelson, M., Vershynin, R.: On sparse reconstruction from Fourier and Gaussian measurements. Commun. Pure Appl. Math. 61(8), 1025–1045 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  36. 36.

    Saunders, M., Kim, B.: PDCO: Primal-dual interior method for convex objectives. Technical Report, Stanford University. http://www.stanford.edu/group/SOL/software/pdco.html (2002)

  37. 37.

    Takhar, D., Laska, J.N., Wakin, M., Duarte, M., Baron, D., Sarvotham, S., Kelly, K. K., Baraniuk, R.G.: A new camera architecture based on optical-domain compression. In: Proceedings of the IS &T/SPIE Symposium on Electronic Imaging: Computational, Imaging, 6065 (2006)

  38. 38.

    Thomson, A.: Compressive single-pixel imaging. In: Proceedings of the 2nd IMA Conference on Mathematics in Defence, Defence Academy, Shrivenham, UK (2011)

  39. 39.

    Wen, Z., Yin, W., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation. SIAM J. Sci. Comput. 32(4), 1809–1831 (2010)

    Google Scholar 

  40. 40.

    Wright, S.J.: Primal–dual interior-point methods. SIAM, Philadelphia (1997)

    Book  MATH  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Kimon Fountoulakis.

Additional information

J. Gondzio is supported by EPSRC Grant EP/I017127/1.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fountoulakis, K., Gondzio, J. & Zhlobich, P. Matrix-free interior point method for compressed sensing problems. Math. Prog. Comp. 6, 1–31 (2014). https://doi.org/10.1007/s12532-013-0063-6

Download citation

Keywords

  • Matrix-free interior point
  • Preconditioned conjugate gradient
  • Compressed sensing
  • Compressive sampling
  • \(\ell _1\)-regularization

Mathematics Subject Classification (2000)

  • 90C05
  • 90C06
  • 90C30
  • 90C25
  • 90C51