Skip to main content

A Lagrange–Newton algorithm for sparse nonlinear programming

Abstract

The sparse nonlinear programming (SNP) problem has wide applications in signal and image processing, machine learning and finance, etc. However, the computational challenge posed by SNP has not yet been well resolved due to the nonconvex and discontinuous \(\ell _0\)-norm involved. In this paper, we resolve this numerical challenge by developing a fast Newton-type algorithm. As a theoretical cornerstone, we establish a first-order optimality condition for SNP based on the concept of strong \(\beta \)-Lagrangian stationarity via the Lagrangian function, and reformulate it as a system of nonlinear equations called the Lagrangian equations. The nonsingularity of the corresponding Jacobian is discussed, based on which the Lagrange–Newton algorithm (LNA) is then proposed. Under mild conditions, we establish the locally quadratic convergence and its iterative complexity estimation. To further demonstrate the efficiency and superiority of our proposed algorithm, we apply LNA to two specific problems arising from compressed sensing and sparse high-order portfolio selection, in which significant benefits accrue from the restricted Newton step.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

Notes

  1. 1.

    HTP is available at:https://github.com/foucart/HTP.

  2. 2.

    NIHT, GP and OMP are available at https://www.southampton.ac.uk/engineering/about/staff/tb1m08.page#software. We use the version sparsity_0_5 in which NIHT, GP and OMP are called hard_l0_Mterm, greed_gp and greed_omp.

  3. 3.

    CoSaMP and SP are available at:http://media.aau.dk/null_space_pursuits/2011/07/a-few-corrections-to-cosamp-and-sp-matlab.html.

  4. 4.

    http://cran.r-project.org/web/packages/portfolioBacktest/vignettes.

  5. 5.

    http://www.mathworks.com/matlabcentral/fileexchange/47839-co_moments-m.

References

  1. 1.

    Beck, A., Eldar, Y.C.: Sparsity constrained nonlinear optimization: optimality conditions and algorithms. SIAM J. Optim. 23(3), 1480–1509 (2013)

    MathSciNet  Article  Google Scholar 

  2. 2.

    Beck, A., Hallak, N.: On the minimization over sparse symmetric sets: projections, optimality conditions, and algorithms. Math. Oper. Res. 41(1), 196–223 (2015)

    MathSciNet  Article  Google Scholar 

  3. 3.

    Beck, A., Vaisbourd, Y.: The sparse principal component analysis problem: Optimality conditions and algorithms. J. Optim. Theory Appl. 170(1), 119–143 (2016)

    MathSciNet  Article  Google Scholar 

  4. 4.

    Blumensath, T., Davies, M.E.: Gradient pursuits. IEEE Trans. Signal Process. 56(6), 2370–2382 (2008)

    MathSciNet  Article  Google Scholar 

  5. 5.

    Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009)

    MathSciNet  Article  Google Scholar 

  6. 6.

    Blumensath, T., Davies, M.E.: Normalized iterative hard thresholding: guaranteed stability and performance. IEEE J. Sel. Top. Signal Process. 4(2), 298–309 (2010)

    Article  Google Scholar 

  7. 7.

    Boudt, K., Lu, W., Peeters, B.: Higher order comoments of multifactor models and asset allocation. Financ. Res. Lett. 13, 225–233 (2015)

    Article  Google Scholar 

  8. 8.

    Červinka, M., Kanzow, C., Schwartz, A.: Constraint qualifications and optimality conditions for optimization problems with cardinality constraints. Math. Program. 160(1), 353–377 (2016)

    MathSciNet  Article  Google Scholar 

  9. 9.

    Chen, J., Gu, Q.: Fast Newton hard thresholding pursuit for sparsity constrained nonconvex optimization. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 757–766 (2017)

  10. 10.

    Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(l_2-l_p\) minimization. Math. Program. 143(1–2), 371–383 (2014)

    MathSciNet  Article  Google Scholar 

  11. 11.

    Dai, W., Milenkovic, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 55(5), 2230–2249 (2009)

    MathSciNet  Article  Google Scholar 

  12. 12.

    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    MathSciNet  Article  Google Scholar 

  13. 13.

    Elad, M.: Sparse and Redundant Representations. Springer, New York (2010)

    Book  Google Scholar 

  14. 14.

    Elad, M., Figueiredo, M.A., Ma, Y.: On the role of sparse and redundant representations in image processing. Proc. IEEE 98(6), 972–982 (2010)

    Article  Google Scholar 

  15. 15.

    Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its Oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)

    MathSciNet  Article  Google Scholar 

  16. 16.

    Foucart, S.: Hard thresholding pursuit: an algorithm for compressive sensing. SIAM J. Numer. Anal. 49(6), 2543–2563 (2011)

    MathSciNet  Article  Google Scholar 

  17. 17.

    Gao, J., Li, D.: Optimal cardinality constrained portfolio selection. Oper. Res. 61(3), 745–761 (2013)

    MathSciNet  Article  Google Scholar 

  18. 18.

    Gotoh, J.Y., Takeda, A., Tono, K.: DC formulations and algorithms for sparse optimization problems. Math. Program 169(1), 141–176 (2018)

    MathSciNet  Article  Google Scholar 

  19. 19.

    Han, S.P.: Superlinearly convergent variable metric algorithms for general nonlinear programming problems. Math. Program. 11, 263–282 (1976)

    MathSciNet  Article  Google Scholar 

  20. 20.

    Koh, K., Kim, S.J., Boyd, S.: An interior-point method for large-scale \(\ell _1\)-regularized logistic regression. J. Mach. Learn. Res. 8, 1519–1555 (2007)

    MathSciNet  MATH  Google Scholar 

  21. 21.

    Kyrillidis, A., Becker, S., Cevher, V., Koch, C.: Sparse projections onto the simplex. In: Proceedings of the 30th international conference on machine learning, Atlanta, Georgia, USA, 2013, vol. 28, pp. 235–243 (2013)

  22. 22.

    Lu, Z.: Optimization over sparse symmetric sets via a nonmonotone projected gradient method. arXiv:1509.08581 (2015)

  23. 23.

    Lu, Z., Zhang, Y.: Sparse approximation via penalty decomposition methods. SIAM J. Optim. 23(4), 2448–2478 (2013)

    MathSciNet  Article  Google Scholar 

  24. 24.

    Luo, Z., Sun, D., Toh, K.C., Xiu, N.: Solving the OSCAR and SLOPE models using a semismooth Newton-based augmented Lagrangian method. J. Mach. Learn. Res. 20(106), 1–25 (2019)

    MathSciNet  MATH  Google Scholar 

  25. 25.

    Luo, Z.Q., Pang, J.S., Ralph, D. Piecewise sequential quadratic programming for mathematical programs with nonlinear complementarity constraints. In: Migdalas, A., Pardalos, P.M., Värbrand, P. (eds.) Multilevel Optimization: Algorithms and Applications. Nonconvex Optimization and Its Applications, vol 20. Springer, Boston, MA (1998)

  26. 26.

    Misra, J.: Interactive exploration of microarray gene expression patterns in a reduced dimensional space. Genome Res. 12(7), 1112–1120 (2002)

    Article  Google Scholar 

  27. 27.

    Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)

    MathSciNet  Article  Google Scholar 

  28. 28.

    Needell, D., Tropp, J.A.: CoSaMP: iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26(3), 301–321 (2009)

    MathSciNet  Article  Google Scholar 

  29. 29.

    Negahban, S.N., Ravikumar, P., Wainwright, M.J., Yu, B., et al.: A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers. Stat. Sci. 27(4), 538–557 (2012)

    MathSciNet  Article  Google Scholar 

  30. 30.

    Pan, L., Luo, Z., Xiu, N.: Restricted Robinson constraint qualification and optimality for cardinality-constrained cone programming. J. Optim. Theory Appl. 175(1), 104–118 (2017)

    MathSciNet  Article  Google Scholar 

  31. 31.

    Pan, L., Xiu, N., Fan, J.: Optimality conditions for sparse nonlinear programming. Sci. China Math. 60(5), 759–776 (2017)

    MathSciNet  Article  Google Scholar 

  32. 32.

    Pan, L., Xiu, N., Zhou, S.: On solutions of sparsity constrained optimization. J. Oper. Res. Soc. China 3(4), 421–439 (2015)

    MathSciNet  Article  Google Scholar 

  33. 33.

    Pan, L., Zhou, S., Xiu, N., Qi, H.D.: Convergent iterative hard thresholding for sparsity and nonnegativity constrained optimization. Pacif. J. Optim. 13(2), 325–353 (2017)

    MATH  Google Scholar 

  34. 34.

    Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In: Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, pp. 40–44. IEEE (1993)

  35. 35.

    Powell, M.J.D.: Restart procedures for the conjugate gradient method. Math. Program. 12, 241–254 (1977)

    MathSciNet  Article  Google Scholar 

  36. 36.

    Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inf. Theory 53(12), 4655–4666 (2007)

    MathSciNet  Article  Google Scholar 

  37. 37.

    Wang, J., Deng, Z., Zheng, T., So, A.M.C.: Sparse high-order portfolios via proximal DCA and SCA. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5425–5429 (2021)

  38. 38.

    Wilson, R.B.: A simplicial algorithm for concave programming. Ph.D. thesis, Graduate School of Business Administration, Harvard University (1963)

  39. 39.

    Xu, F., Lu, Z., Xu, Z.: An efficient optimization approach for a cardinality-constrained index tracking problem. Optim. Methods Softw. 31, 258–271 (2016)

    MathSciNet  Article  Google Scholar 

  40. 40.

    Yin, P., Lou, Y., He, Q., Xin, J.: Minimization of \(\ell _{1-2}\) for compressed sensing. SIAM J. Sci. Comput. 37(1), 536–563 (2015)

    MathSciNet  Article  Google Scholar 

  41. 41.

    Yuan, X., Li, P., Zhang, T.: Gradient hard thresholding pursuit. J. Mach. Learn. Res. 18(166), 1–43 (2018)

    MathSciNet  MATH  Google Scholar 

  42. 42.

    Yuan, X., Liu, Q.: Newton greedy pursuit: a quadratic approximation method for sparsity-constrained optimization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 4122–4129 (2014)

  43. 43.

    Yuan, X., Liu, Q.: Newton-type greedy selection methods for \(\ell _0\)-constrained minimization. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2437–2450 (2017)

    Article  Google Scholar 

  44. 44.

    Zhou, S., Xiu, N., Qi, H.: Global and quadratic convergence of Newton hard-thresholding pursuit. J. Mach. Learn. Res. 22, 1–45 (2021)

    MathSciNet  MATH  Google Scholar 

  45. 45.

    Zhou, S., Xiu, N., Wang, Y., Kong, L., Qi, H.D.: A null-space-based weighted \(\ell _1\) minimization approach to compressed sensing. Inf. Inference J. IMA 5(1), 76–102 (2016)

    Article  Google Scholar 

  46. 46.

    Zhou, T., Tao, D., Wu, X.: Manifold elastic net: a unified framework for sparse dimension reduction. Data Min. Knowl. Disc. 22(3), 340–371 (2011)

    MathSciNet  Article  Google Scholar 

  47. 47.

    Zou, H., Hastie, T., Tibshirani, R.: Sparse principal component analysis. J. Comput. Graph. Stat. 15(2), 265–286 (2006)

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

We would like to thank AE and the two referees for their valuable comments, which have helped to shorten the paper and greatly improved its presentation. We also thank Dr. Shenglong Zhou of Imperial College, London for his great support on the numerical experiments.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ziyan Luo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was partially supported by the National Natural Science Foundation of China (11771038, 11971052, 12011530155) and Beijing Natural Science Foundation (Z190002)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhao, C., Xiu, N., Qi, H. et al. A Lagrange–Newton algorithm for sparse nonlinear programming. Math. Program. (2021). https://doi.org/10.1007/s10107-021-01719-x

Download citation

Keywords

  • Sparse nonlinear programming
  • Lagrangian equation
  • The Newton method
  • Locally quadratic convergence
  • Application

Mathematics Subject Classification

  • 90C30
  • 49M15
  • 90C46