Accelerated Smoothing Hard Thresholding Algorithms for $\ell _0$ Regularized Nonsmooth Convex Regression Problem

Abstract

We study a class of constrained sparse optimization problems with cardinality penalty, where the feasible set is defined by box constraint, and the loss function is convex but not necessarily smooth. First, we propose an accelerated smoothing hard thresholding (ASHT) algorithm for solving such problems, which combines smoothing approximation, extrapolation technique and iterative hard thresholding method. The extrapolation coefficients can be chosen to satisfy $\sup _k \beta _k=1$. We discuss the convergence of ASHT algorithm with different extrapolation coefficients, and give a sufficient condition to ensure that any accumulation point of the iterates is a local minimizer of the original problem. For a class of special updating schemes on the extrapolation coefficients, we obtain that the iterates are convergent to a local minimizer of the problem, and the convergence rate is $o(\ln ^{\sigma } k/k)$ with $\sigma \in (1/2, 1]$ on the loss and objective function values. Second, we consider the case in which the loss function is Lipschitz continuously differentiable, and develop an accelerated hard thresholding (AHT) algorithm to solve it. We prove that the iterates of AHT algorithm converge to a local minimizer of the problem that satisfies a desirable lower bound property. Moreover, we show that the convergence rates of loss and objective function values are $o(k^{-2})$. Finally, some numerical examples are presented to show the theoretical results.