Advertisement

Computational Optimization and Applications

, Volume 70, Issue 2, pp 443–478 | Cite as

A primal-dual homotopy algorithm for \(\ell _{1}\)-minimization with \(\ell _{\infty }\)-constraints

  • Christoph Brauer
  • Dirk A. Lorenz
  • Andreas M. Tillmann
Article

Abstract

In this paper we propose a primal-dual homotopy method for \(\ell _1\)-minimization problems with infinity norm constraints in the context of sparse reconstruction. The natural homotopy parameter is the value of the bound for the constraints and we show that there exists a piecewise linear solution path with finitely many break points for the primal problem and a respective piecewise constant path for the dual problem. We show that by solving a small linear program, one can jump to the next primal break point and then, solving another small linear program, a new optimal dual solution is calculated which enables the next such jump in the subsequent iteration. Using a theorem of the alternative, we show that the method never gets stuck and indeed calculates the whole path in a finite number of steps. Numerical experiments demonstrate the effectiveness of our algorithm. In many cases, our method significantly outperforms commercial LP solvers; this is possible since our approach employs a sequence of considerably simpler auxiliary linear programs that can be solved efficiently with specialized active-set strategies.

Keywords

Convex optimization Dantzig selector Homotopy methods Nonsmooth optimization Primal-dual methods 

Mathematics Subject Classification

90C05 90C25 65K05 

Supplementary material

References

  1. 1.
    Appa, G., Smith, C.: On \(L_{1}\) and Chebyshev estimation. Math. Program. 5, 73–87 (1973).  https://doi.org/10.1007/BF01580112 CrossRefMATHGoogle Scholar
  2. 2.
    Asif, M.S.: Primal dual pursuit—a homotopy based algorithm for the Dantzig selector. Master’s thesis, Georgia Institute of Technology (2008)Google Scholar
  3. 3.
    Asif, M.S., Romberg, J.: Dantzig selector homotopy with dynamic measurements. In: Proceedings on SPIE 7246, Computational Imaging VII, 72460E (2009)Google Scholar
  4. 4.
    Asif, M.S., Romberg, J.: On the Lasso and Dantzig selector equivalence. In: Proceedings on CISS. IEEE (2010)Google Scholar
  5. 5.
    Blum, L.: A new simple homotopy algorithm for linear programming I. J. Complex. 4, 124–136 (1988)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Brauer, C., Gerkmann, T., Lorenz, D.A.: Sparse reconstruction of quantized speech signals. In: Proceedings on ICASSP. IEEE (2016)Google Scholar
  7. 7.
    Cai, T., Liu, W.: A direct estimation approach to sparse linear discriminant analysis. J. Am. Stat. Assoc. 106(496), 1566–1577 (2011)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Cai, T., Liu, W., Luo, X.: A constrained \(\ell _1\) minimization approach to sparse precision matrix estimation. J. Am. Stat. Assoc. 106(494), 594–607 (2011)CrossRefMATHGoogle Scholar
  9. 9.
    Candés, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Candés, E.J., Tao, T.: The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). Ann. Stat. 35(6), 2313–2351 (2007)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Dantzig, G.B.: Linear Programming and Extensions. Princeton University Press, Princeton (1963)CrossRefMATHGoogle Scholar
  13. 13.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Eldar, Y.C., Kutyniok, G. (eds.): Compressed Sensing: Theory and Applications. Cambridge University Press, Cambridge (2012)Google Scholar
  15. 15.
    Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Birkhäuser, Basel (2013)CrossRefMATHGoogle Scholar
  16. 16.
    Jacques, L., Hammond, D.K., Fadili, J.M.: Dequantizing compressed sensing: when oversampling and non-Gaussian constraints combine. IEEE Trans. Inf. Theory 57(1), 559–571 (2011)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    James, G.M., Radchenko, P., Lv, J.: DASSO: connections between the Dantzig selector and Lasso. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 71(1), 127–142 (2009)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: International Joint Conference on Artificial Intelligence (IJCAI) (1995)Google Scholar
  19. 19.
    Lorenz, D.A., Pfetsch, M.E., Tillmann, A.M.: Solving basis pursuit: heuristic optimality check and solver comparison. ACM Trans. Math. Softw. 41(2), 8 (2015)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Mairal, J., Yu, B.: Complexity analysis of the lasso regularization path. ArXiv preprint arXiv:1205.0079 (2012)
  21. 21.
    Nazareth, J.L.: The homotopy principle and algorithms for linear programming. SIAM J. Optim. 1(3), 316–332 (1991)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, Berlin (2006)MATHGoogle Scholar
  23. 23.
    Osborne, M.R., Presnell, B., Turlach, B.A.: A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20, 389–404 (2000)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Pang, H., Zhao, T., Vanderbei, R.J., Liu, H.: A parametric simplex approach to statistical learning problems. Unpublished manuscript. http://www.princeton.edu/~rvdb/tex/PSM/PSM.pdf (2015)
  25. 25.
    Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)CrossRefMATHGoogle Scholar
  26. 26.
    Schrijver, A.: Theory of Linear and Integer Programming. Wiley, Hoboken (1986)MATHGoogle Scholar
  27. 27.
    Späth, H.: Mathematical Algorithms for Linear Regression. Computer Science and Scientific Computing. Translated and Revised from the 1987 German Original by the Author. Academic Press Inc., Boston (1992)MATHGoogle Scholar
  28. 28.
    Stiefel, E.: Über diskrete und lineare tschebyscheff-approximationen. Numerische Mathematik 1(1), 1–28 (1959).  https://doi.org/10.1007/BF01386369 MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Vanderbei, R.J.: Linear Programming: Foundations and Extensions, 2nd edn. Kluwer Academic Publishers, Dordrecht (2001)CrossRefMATHGoogle Scholar
  30. 30.
    Zheng, S., Liu, W.: An experimental comparison of gene selection by Lasso and Dantzig selector for cancer classification. Comput. Biol. Med. 41(11), 1033–1040 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Institut für Analysis und AlgebraTechnische Universität BraunschweigBraunschweigGermany
  2. 2.Visual Computing Institute & Chair of Operations ResearchRWTH Aachen UniversityAachenGermany

Personalised recommendations