Calculus of the Exponent of Kurdyka–Łojasiewicz Inequality and Its Applications to Linear Convergence of First-Order Methods

Abstract

In this paper, we study the Kurdyka–Łojasiewicz (KL) exponent, an important quantity for analyzing the convergence rate of first-order methods. Specifically, we develop various calculus rules to deduce the KL exponent of new (possibly nonconvex and nonsmooth) functions formed from functions with known KL exponents. In addition, we show that the well-studied Luo–Tseng error bound together with a mild assumption on the separation of stationary values implies that the KL exponent is $\frac{1}{2}$. The Luo–Tseng error bound is known to hold for a large class of concrete structured optimization problems, and thus we deduce the KL exponent of a large class of functions whose exponents were previously unknown. Building upon this and the calculus rules, we are then able to show that for many convex or nonconvex optimization models for applications such as sparse recovery, their objective function’s KL exponent is $\frac{1}{2}$. This includes the least squares problem with smoothly clipped absolute deviation regularization or minimax concave penalty regularization and the logistic regression problem with $\ell _1$ regularization. Since many existing local convergence rate analysis for first-order methods in the nonconvex scenario relies on the KL exponent, our results enable us to obtain explicit convergence rate for various first-order methods when they are applied to a large variety of practical optimization models. Finally, we further illustrate how our results can be applied to establishing local linear convergence of the proximal gradient algorithm and the inertial proximal algorithm with constant step sizes for some specific models that arise in sparse recovery.

Appendix: An Auxiliary Lemma

In this appendix, we prove a version of [41, Lemma 6] for a class of proper closed functions taking the form $f:= \ell +P$, where $\ell$ is a proper closed function with an open domain and is continuously differentiable on $\mathrm{dom}\,\ell$, and P is a proper closed polyhedral function. Our proof follows exactly the same line of arguments as [41, Lemma 6] and is only included here for the sake of completeness.

In what follows, we let $K:= \{(x,s):\;s\ge P(x)\}$ and define

$$\begin{aligned} g(x,s) = \underbrace{\ell (x) + s}_{h(x,s)} + \delta _K(x,s). \end{aligned}$$

Then we have the following result.

Lemma A.1

There exists $C > 0$ so that for any $x\in \mathrm{dom}\,f$, we have

$$\begin{aligned} \Vert \mathrm{Proj}_K[(x,P(x)) - \nabla h(x,P(x))] - (x,P(x))\Vert \le C \Vert \mathrm{prox}_P(x-\nabla \ell (x)) - x\Vert . \end{aligned}$$

Proof

For notational simplicity, let

$$\begin{aligned} \begin{aligned} (y,\mu )&:= \mathrm{Proj}_K[(x,P(x)) - \nabla h(x,P(x))],\\ w&:= \mathrm{prox}_P(x-\nabla \ell (x)). \end{aligned} \end{aligned}$$

Note that $\nabla h(x,P(x)) = (\nabla \ell (x),1)$. Using these and the definitions of proximal mapping and projection, we have

$$\begin{aligned} (y,\mu )&= \mathop {\mathrm{arg\,min}}\limits _{(u,s)\in K}\left\{ \langle \nabla \ell (x),u-x\rangle + (s - P(x)) + \frac{1}{2}\Vert u-x\Vert ^2 + \frac{1}{2}(s - P(x))^2\right\} , \end{aligned}$$

(40)

$$\begin{aligned} w&= \mathop {\mathrm{arg\,min}}\limits _{u}\left\{ \langle \nabla \ell (x),u-x\rangle + \frac{1}{2}\Vert u-x\Vert ^2 +P(u)\right\} . \end{aligned}$$

(41)

Now, using the strong convexity of the objective function in (40) and comparing its function values at the points $(y,\mu )$ and (w, P(w)), we have

$$\begin{aligned} \begin{aligned}&\langle \nabla \ell (x),y-x\rangle + (\mu - P(x)) + \frac{1}{2}\Vert y-x\Vert ^2 + \frac{1}{2}(\mu - P(x))^2\\&\le \langle \nabla \ell (x),w-x\rangle + (P(w) - P(x)) + \frac{1}{2}\Vert w-x\Vert ^2 + \frac{1}{2}(P(w) - P(x))^2\\&\ \ \ \ - \frac{1}{2} \Vert y - w\Vert ^2 - \frac{1}{2} (\mu - P(w))^2. \end{aligned} \end{aligned}$$

(42)

Similarly, using the strong convexity of the objective function in (41) and comparing its function values at the points w and y, we have

$$\begin{aligned} \begin{aligned}&\langle \nabla \ell (x),w-x\rangle + \frac{1}{2}\Vert w-x\Vert ^2 +P(w)\\&\le \langle \nabla \ell (x),y-x\rangle + \frac{1}{2}\Vert y-x\Vert ^2 +P(y) - \frac{1}{2} \Vert y - w\Vert ^2\\&\le \langle \nabla \ell (x),y-x\rangle + \frac{1}{2}\Vert y-x\Vert ^2 +\mu - \frac{1}{2} \Vert y - w\Vert ^2, \end{aligned} \end{aligned}$$

(43)

where the last inequality follows from the fact that $(y,\mu )\in K$. Summing the inequalities (42) and (43) and rearranging terms, we see further that

$$\begin{aligned} \frac{1}{2}(\mu - P(x))^2 + \Vert y - w\Vert ^2\le \frac{1}{2}(P(w)-P(x))^2 - \frac{1}{2}(\mu - P(w))^2. \end{aligned}$$

(44)

Since P is a proper closed polyhedral function, it is piecewise linear on its domain (see, e.g., [8, Proposition 5.1.1]) and hence is Lipschitz continuous on its domain. Thus, it follows from this and (44) that there exists $M > 0$ so that

$$\begin{aligned} |\mu - P(x)|&\le |P(w) - P(x)| \le M \Vert w - x\Vert \ \ \mathrm{and}\nonumber \\&\Vert y - w\Vert \le |P(w) - P(x)|\le M\Vert w-x\Vert . \end{aligned}$$

(45)

Moreover, we can deduce further from the second relation in (45) that

$$\begin{aligned} \Vert y - x\Vert \le \Vert y - w\Vert + \Vert w - x\Vert \le (M+1)\Vert w-x\Vert . \end{aligned}$$

This together with the first relation in (45) and the definitions of $(y,\mu )$ and w completes the proof.$\square$