Skip to main content
SpringerLink
Log in

Weighted thresholding homotopy method for sparsity constrained optimization

  • Published:
Journal of Combinatorial Optimization Aims and scope Submit manuscript

Abstract

We propose in this paper a novel weighted thresholding method for the sparsity-constrained optimization problem. By reformulating the problem equivalently as a mixed-integer programming, we investigate the Lagrange duality with respect to an \(l_1\)-norm constraint and show the strong duality property. Then we derive a weighted thresholding method for the inner Lagrangian problem, and analyze its convergence. In addition, we give an error bound of the solution under some assumptions. Further, based on the proposed method, we develop a homotopy algorithm with varying sparsity level and Lagrange multiplier, and prove that the algorithm converges to an L-stationary point of the primal problem under some conditions. Computational experiments show that the proposed algorithm is competitive with state-of-the-art methods for the sparsity-constrained optimization problem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. MATLAB packages for NIHT and AIHT: http://www.personal.soton.ac.uk/tb1m08/sparsify/sparsify.html.

  2. MATLAB packages for ECME and DORE: http://home.eng.iastate.edu/~ald/DORE.htm.

  3. MATLAB package for GraSP: http://sbahmani.ece.gatech.edu/GraSP.html.

References

  • Bahmani S, Raj B, Boufounos P (2013) Greedy sparsity-constrained optimization. J Mach Learn Res 14(1):807–841

    MathSciNet  MATH  Google Scholar 

  • Beck A, Hallak N (2016) On the minimization over sparse symmetric sets: projections, optimality conditions, and algorithms. Math Oper Res 41(1):196–223

    Article  MathSciNet  Google Scholar 

  • Beck A, Teboulle M (2009) A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J Imaging Sci 2(1):183–202

    Article  MathSciNet  Google Scholar 

  • Bi S, Liu X, Pan S (2014) Exact Penalty Decomposition Method for Zero-Norm Minimization Based on MPEC Formulation. SIAM J Sci Comput 36(4):A1451–A1477

    Article  MathSciNet  Google Scholar 

  • Blumensath T (2012) Accelerated iterative hard thresholding. Sig Process 92(3):752–756

    Article  Google Scholar 

  • Blumensath T, Davies ME (2008) Iterative thresholding for sparse approximations. J Fourier Anal Appl 14(5–6):629–654

    Article  MathSciNet  Google Scholar 

  • Blumensath T, Davies ME (2010) Normalised itertive hard thresholding: guaranteed stability and performance. IEEE J Sel Top Signal Process 4(2):298–309

    Article  Google Scholar 

  • Candès EJ, Tao T (2005) Decoding by linear programming. IEEE Trans Inf Theory 51(12):4203–4215

    Article  MathSciNet  Google Scholar 

  • Candès EJ, Wakin MB, Boyd SP (2008) Enhancing sparsity by reweighted \(l_1\) minimization. J Fourier Anal Appl 14(5–6):877–905

    Article  MathSciNet  Google Scholar 

  • Chen Y, Ye Y, Wang M (2019) Approximation hardness for a class of sparse optimization problems. J Mach Learn Res 20:1–27

    MathSciNet  MATH  Google Scholar 

  • Dong Z, Zhu W (2018) Homotopy methods based on \(l_0\) norm for the compressed sensing problem. IEEE Trans Neural Netw Learn Syst 29(4):1132–1146

    Article  Google Scholar 

  • Donoho DL, Tsaig Y (2008) Fast solution of \(l_1\)-norm minimization problems when the solution may be sparse. IEEE Trans Inf Theory 54(11):4789–4812

    Article  Google Scholar 

  • Elad M (2010) Sparse and redundant representations: from theory to applications in signal and image processing. Springer, New York

    Book  Google Scholar 

  • Fan J, Li R (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc 96(456):1348–1360

    Article  MathSciNet  Google Scholar 

  • Friedman J, Hastie T, Tibshirani R (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33(1):1–22

    Article  Google Scholar 

  • Guyon I, Gunn S, Ben-Hur A, Dror G (2005) Result analysis of the nips 2003 feature selection challenge. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing system 17. MIT-Press, Cambridge, MA, pp 545–552

    Google Scholar 

  • Jain P, Rao N, Dhillon I (2016) Structured sparse regression via greedy hard-thresholding. In: Advances in neural information processing systems (NIPS), pp 1516–1524

  • Jiao Y, Jin B, Lu X (2015) A primal dual active set with continuation algorithm for the \(l_0\)-regularized optimization problem. Appl Comput Harmon Anal 39(3):400–426

    Article  MathSciNet  Google Scholar 

  • Khajehnejad MA, Xu W, Avestimehr AS, Hassibi B (2009) Weighted \(l_1\) minimization for sparse recovery with prior information. In 2009 IEEE international conference on symposium on information theory, pp 483–487

  • Koh K, Kim S, Boyd S (2007) An interior-point method for large-scale \(l_1\)-regularized logistic regression. J Mach Learn Res 8:1519–1555

    MathSciNet  MATH  Google Scholar 

  • Li Q, Bai Y, Yu C, Yuan Y-X (2019) A new piecewise quadratic approximation approach for \(l_0\) norm minimization problem. Sci China Math 62(1):185–204

    Article  MathSciNet  Google Scholar 

  • Li X, Zhao T, Arora R, Liu H, Haupt J (2016) Stochastic variance reduced optimization for nonconvex sparse learning. In: International conference on machine learning (ICML), pp 917–925

  • Liu B, Yuan X, Wang L, Liu Q, Metaxas DN (2017) Dual iterative hard thresholding: from non-convex sparse minimization to non-smooth concave maximization. In: International conference on machine learning (ICML), pp 2179–2187

  • Liu Y, Bi S, Pan S (2018) Equivalent Lipschitz surrogates for zero-norm and rank optimization problems. J Global Optim 72(4):679–704

    Article  MathSciNet  Google Scholar 

  • Lu Z (2014) Iterative hard thresholding methods for \(l_0\) regularized convex cone programming. Math Program 147(1–2):125–154

    Article  MathSciNet  Google Scholar 

  • Lu Z, Zhang Y (2013) Sparse approximation via penalty decomposition methods. SIAM J Optim 23(4):2448–2478

    Article  MathSciNet  Google Scholar 

  • Mairal J, Bach F, Ponce J (2014) Sparse modeling for image and vision processing. Found Trends Comput Graphics Vis 8(2–3):85–283

    Article  Google Scholar 

  • Mallat S, Zhang Z (1993) Matching pursuits with time-frequency dictionaries. IEEE Trans Signal Process 41(12):3397–3415

    Article  Google Scholar 

  • Needell D, Tropp JA (2009) Cosamp: iterative signal recovery from incomplete and inaccurate samples. Appl Comput Harmon Anal 26(3):301–321

    Article  MathSciNet  Google Scholar 

  • Newman D, Hrttich S, Blake C, Merz C (1998) UCI repository of machine learing databases. www.ics.uci.edu/~mlearn/MLRepository.html

  • Nguyen N, Needell D, Woolf T (2017) Linear convergence of stochastic iterative greedy algorithms with sparse constraints. IEEE Trans Inf Theory 63(11):6869–6895

    Article  MathSciNet  Google Scholar 

  • Qiu K, Dogandžić A (2012) Sparse signal reconstruction via ecme hard thresholding. IEEE Trans Signal Process 60(9):4551–4569

    Article  MathSciNet  Google Scholar 

  • Rakotomamonjy A, Koco S, Ralaivola L (2017) Greedy methods, randomization approaches, and multiarm bandit algorithms for efficient sparsity-constrained optimization. IEEE Trans Neural Netw Learn Syst 28(11):2789–2802

    Article  MathSciNet  Google Scholar 

  • Shen X, Pan W, Zhu Y, Zhou H (2013) On constrained and regularized high-dimensional regression. Ann Inst Stat Math 65(5):807–832

    Article  MathSciNet  Google Scholar 

  • Soussen C, Idier J, Duan J, Brie D (2015) Homotopy based algorithms for \(l_0\)-regularized least-squares. IEEE Trans Signal Process 63(13):3301–3316

    Article  MathSciNet  Google Scholar 

  • Tropp JA, Gilbert AC (2007) Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans Inf Theory 53(12):4655–4666

    Article  MathSciNet  Google Scholar 

  • Xiang S, Shen X, Ye J (2015) Efficient nonconvex sparse group feature selection via continuous and discrete optimization. Artif Intell 224:28–50

    Article  MathSciNet  Google Scholar 

  • Xiao L, Zhang T (2013) A proximal-gradient homotopy method for the sparse least-squares problem. SIAM J Optim 23(2):1062–1091

    Article  MathSciNet  Google Scholar 

  • Xu Z, Chang X, Xu F, Zhang H (2012) \(l_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans Neural Netw Learn Syst 23(7):1013–1027

    Article  Google Scholar 

  • Yuan X, Li P, Zhang T (2018) Gradient hard thresholding pursuit. J Mach Learn Res 18:1–43

    MathSciNet  MATH  Google Scholar 

  • Zhang C-H, Zhang T (2012) A general theory of concave regularization for high-dimensional sparse estimation problems. Stat Sci 27(4):576–593

    Article  MathSciNet  Google Scholar 

  • Zhang T (2010) Analysis of multi-stage convex relaxation for sparse regularization. J Mach Learn Res 11:1081–1107

    MathSciNet  MATH  Google Scholar 

  • Zhang Z, Xu Y, Yang J, Li X, Zhang D (2015) A survey of sparse representation: algorithms and applications. IEEE Access 3:490–530

    Article  Google Scholar 

  • Zhao Y (2018) Sparse optimization theory and methods. CRC Press/Taylor and Francis Group, Boca Raton

    Book  Google Scholar 

  • Zhao Y, Li D (2012) Reweighted \(l_1\)-minimization for sparse solutions to underdetermined linear systems. SIAM J Optim 22(3):1065–1088

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their expertise comments on our manuscript, which have helped improving the quality and clarity of this paper.

Author information

Authors and Affiliations

  1. Center for Discrete Mathematics and Theoretical Computer Science, Fuzhou University, Fuzhou, 350116, China

    Wenxing Zhu, Huating Huang, Lanfan Jiang & Jianli Chen

Authors
  1. Wenxing Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huating Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lanfan Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianli Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lanfan Jiang.

Additional information

Dedicated to Professor Minyi Yue on the Occasion of his 100th Birthday.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was supported by the National Natural Science Foundation of China under Grant 61672005.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, W., Huang, H., Jiang, L. et al. Weighted thresholding homotopy method for sparsity constrained optimization. J Comb Optim 44, 1924–1952 (2022). https://doi.org/10.1007/s10878-020-00563-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10878-020-00563-7

Keywords

Search

Navigation