Mathematical Programming Computation

, Volume 6, Issue 1, pp 1–31 | Cite as

Matrix-free interior point method for compressed sensing problems

  • Kimon FountoulakisEmail author
  • Jacek Gondzio
  • Pavel Zhlobich
Full Length Paper


We consider a class of optimization problems for sparse signal reconstruction which arise in the field of compressed sensing (CS). A plethora of approaches and solvers exist for such problems, for example GPSR, FPC_AS, SPGL1, NestA, \(\mathbf{\ell _1\_\ell _s}\), PDCO to mention a few. CS applications lead to very well conditioned optimization problems and therefore can be solved easily by simple first-order methods. Interior point methods (IPMs) rely on the Newton method hence they use the second-order information. They have numerous advantageous features and one clear drawback: being the second-order approach they need to solve linear equations and this operation has (in the general dense case) an \({\mathcal {O}}(n^3)\) computational complexity. Attempts have been made to specialize IPMs to sparse reconstruction problems and they have led to interesting developments implemented in \(\mathbf{\ell _1\_\ell _s}\) and PDCO softwares. We go a few steps further. First, we use the matrix-free IPM, an approach which redesigns IPM to avoid the need to explicitly formulate (and store) the Newton equation systems. Secondly, we exploit the special features of the signal processing matrices within the matrix-free IPM. Two such features are of particular interest: an excellent conditioning of these matrices and the ability to perform inexpensive (low complexity) matrix–vector multiplications with them. Computational experience with large scale one-dimensional signals confirms that the new approach is efficient and offers an attractive alternative to other state-of-the-art solvers.


Matrix-free interior point Preconditioned conjugate gradient Compressed sensing Compressive sampling \(\ell _1\)-regularization 

Mathematics Subject Classification (2000)

90C05 90C06 90C30 90C25 90C51 


  1. 1.
    Ahmed, N., Rao, K.R.: Orthogonal transforms for digital signal processing. Springer, Berlin (1975)CrossRefzbMATHGoogle Scholar
  2. 2.
    Baraniuk, R., Davenport, M., DeVore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28(3), 253–263 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Becker, S., Bobin, J., Candés, E.J.: Nesta: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4(1), 1–39 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  4. 4.
    Van Den Berg, E., Friedlander, M.P.: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31(2), 890–912 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Van Den Berg, E., Friedlander, M.P., Hennenfent, G., Herrman, F.J., Saab, R., Yılmaz, Ö.: Sparco: a testing framework for sparse reconstruction. ACM Trans. Math. Softw. 35(4), 1–16 (2009)CrossRefGoogle Scholar
  6. 6.
    Blanchard, J.D., Cartis, C., Tanner, J.: Decay properties of restricted isometry constants. Comput. Optim. Appl. 16(7), 572–575 (2009)Google Scholar
  7. 7.
    Blanchard, J.D., Cartis, C., Tanner, J.: Compressed sensing: how sharp is the restricted isometry property? SIAM Rev. 53(1), 105–125 (2011)Google Scholar
  8. 8.
    Candés, E., Demanet, L., Donoho, D., Ying, L.: Curvelab.
  9. 9.
    Candés, E.J.: Compressive sampling. In: Proceedings of the International Congress of Mathematicians (2006)Google Scholar
  10. 10.
    Candés, E.J., Romberg, J.: Practical signal recovery from random projections. In: Proceedings of the SPIE Conference on Wavelet Applications in Signal and Image Processing XI, 5914 (2004)Google Scholar
  11. 11.
    Candés, E.J., Romberg, J.: \(\ell _1\)-magic. Technical Report, Caltech. (2007)
  12. 12.
    Candés, E.J., Romberg, J.: Sparsity and incoherence in compressive sampling. Inverse Probl. 23(3), 969–985 (2007)CrossRefzbMATHGoogle Scholar
  13. 13.
    Candés, E.J., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Comm. Pure Appl. Math. 59(8), 1207–1223 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  14. 14.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Coifman, R., Geshwind, F., Meyer, Y.: Noiselets. Appl. Comput. Harmon. Anal. 10(1), 27–44 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory 47(7), 2845–2862 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Donoho, D.L., Tanner, J.: Precise undersampling theorems. Proc. IEEE 98(6), 913–924 (2010)CrossRefGoogle Scholar
  19. 19.
    Dossal, C., Mallat, S.: Sparse spike deconvolution with minimum scale. Proc. Signal Process. Adapt. Sparse Struct. Represent. 81(3), 123–126 (1994)Google Scholar
  20. 20.
    Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1(4), 586–597 (2007)CrossRefGoogle Scholar
  21. 21.
    Foucart, S.: A note on guaranteed sparse recovery via 1-minimization. Appl. Comput. Harmon. Anal. 29(1), 97–103 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Frigo, M., Johnston, S.G.: The design and implementation of FFTW3. Proc. IEEE 93(2), 216–231 (2005)CrossRefGoogle Scholar
  23. 23.
    Gondzio, J.: Multiple centrality corrections in a primal–dual method for linear programming. Comput. Optim. Appl. 6, 137–156 (1996)CrossRefzbMATHMathSciNetGoogle Scholar
  24. 24.
    Gondzio, J.: Interior point methods 25 years later. Eur. J. Oper. Res. 218(3), 587–601 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  25. 25.
    Gondzio, J.: Matrix-free interior point method. Comput. Optim. Appl. 51(2), 457–480 (2012)CrossRefzbMATHMathSciNetGoogle Scholar
  26. 26.
    Gondzio, J.: Convergence analysis of an inexact feasible interior point method for convex quadratic programming. SIAM J. Optim. 23(3), 1510–1527 (2013)CrossRefzbMATHMathSciNetGoogle Scholar
  27. 27.
    Hennenfent, G., Herrmann, F.J.: Random sampling: new insights into the reconstruction of coarsely-sampled wavefields. In: SEG International Exposition and 77th Annual Meeting (2007)Google Scholar
  28. 28.
    Kelley, C.T.: Iterative methods for linear and nonlinear equations, volume 16 of Frontiers in Applied Mathematics. SIAM, Philadelphia (1995)Google Scholar
  29. 29.
    Kim, S.-J., Koh, K., Lustig, M., Boyd, S., Gorinevsky, D.: An interior-point method for large-scale \(\ell _1\)-regularized least squares. IEEE J. Sel. Top. Signal Process. 1(4), 606–617 (2007)CrossRefGoogle Scholar
  30. 30.
    Kojima, M., Megiddo, N., Mizuno, S.: A primal-dual infeasible-interior-point algorithm for linear programming. Math. Progr. 61, 263–280 (1993)CrossRefzbMATHMathSciNetGoogle Scholar
  31. 31.
    Lu, Z., Monteiro, R.D.S., O’Neal, J.W.: An iterative solver-based infeasible primal–dual path-following algorithm for convex quadratic programming. SIAM J. Optim. 17, 287–310 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  32. 32.
    Miller, A.J.: Subset selection in regression. Chapmain & Hall/CRC, London (2002)CrossRefzbMATHGoogle Scholar
  33. 33.
    Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)CrossRefzbMATHMathSciNetGoogle Scholar
  34. 34.
    Rockafellar, R.T.: Convex analysis, Princeton landmarks in mathematics and physics. Princeton University Press, USA (1970)Google Scholar
  35. 35.
    Rudelson, M., Vershynin, R.: On sparse reconstruction from Fourier and Gaussian measurements. Commun. Pure Appl. Math. 61(8), 1025–1045 (2008)CrossRefzbMATHMathSciNetGoogle Scholar
  36. 36.
    Saunders, M., Kim, B.: PDCO: Primal-dual interior method for convex objectives. Technical Report, Stanford University. (2002)
  37. 37.
    Takhar, D., Laska, J.N., Wakin, M., Duarte, M., Baron, D., Sarvotham, S., Kelly, K. K., Baraniuk, R.G.: A new camera architecture based on optical-domain compression. In: Proceedings of the IS &T/SPIE Symposium on Electronic Imaging: Computational, Imaging, 6065 (2006)Google Scholar
  38. 38.
    Thomson, A.: Compressive single-pixel imaging. In: Proceedings of the 2nd IMA Conference on Mathematics in Defence, Defence Academy, Shrivenham, UK (2011)Google Scholar
  39. 39.
    Wen, Z., Yin, W., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation. SIAM J. Sci. Comput. 32(4), 1809–1831 (2010)Google Scholar
  40. 40.
    Wright, S.J.: Primal–dual interior-point methods. SIAM, Philadelphia (1997)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2013

Authors and Affiliations

  • Kimon Fountoulakis
    • 1
    Email author
  • Jacek Gondzio
    • 1
  • Pavel Zhlobich
    • 1
  1. 1.School of Mathematics and Maxwell InstituteThe University of EdinburghEdinburgh UK

Personalised recommendations