High-order evaluation complexity for convexly-constrained optimization with non-Lipschitzian group sparsity terms

Abstract

This paper studies high-order evaluation complexity for partially separable convexly-constrained optimization involving non-Lipschitzian group sparsity terms in a nonconvex objective function. We propose a partially separable adaptive regularization algorithm using a pth order Taylor model and show that the algorithm needs at most $O(\epsilon ^{-(p+1)/(p-q+1)})$ evaluations of the objective function and its first p derivatives (whenever they exist) to produce an $(\epsilon ,\delta )$-approximate qth-order stationary point. Our algorithm uses the underlying rotational symmetry of the Euclidean norm function to build a Lipschitzian approximation for the non-Lipschitzian group sparsity terms, which are defined by the group $\ell _2$–$\ell _a$ norm with $a\in (0,1)$. The new result shows that the partially-separable structure and non-Lipschitzian group sparsity terms in the objective function do not affect the worst-case evaluation complexity order.

Appendix

Proof Lemma 3.1

The proof of (3.3) is essentially borrowed from [11, Lemma 2.4], although details differ because the present version covers $a \in (0,1)$. We first observe that $\nabla _\cdot ^j \Vert r\Vert ^a$ is a jth order tensor, whose norm is defined using (1.7). Moreover, using the relationships

$$\begin{aligned} \nabla _\cdot ^1 \Vert r\Vert ^\tau = \tau \, \Vert r\Vert ^{\tau -2}r \;\; \text{ and } \;\; \nabla _\cdot ^1 \big (r^{\tau \otimes }\big ) = \tau \, r^{(\tau -1)\otimes }\otimes I, \;\;\;\;(\tau \in \mathbb {R}), \end{aligned}$$

(A.1)

defining

$$\begin{aligned} \nu _0 {\mathop {=}\limits ^\mathrm{def}}1, \;\; \text{ and } \;\; \nu _i {\mathop {=}\limits ^\mathrm{def}}\prod _{\ell =1}^{i}(a+2-2\ell ), \end{aligned}$$

(A.2)

and proceeding by induction, we obtain that, for some $\mu _{j,i}\ge 0$ with $\mu _{1,1}=1$,

$$\begin{aligned}&\nabla _\cdot ^1\left[ \nabla _\cdot ^{j-1} \Vert r\Vert ^a \right] \\&\quad = \nabla _\cdot ^1\left[ \displaystyle \sum _{i=2}^j \mu _{j-1,i-1} \nu _{i-1} \Vert r\Vert ^{a-2(i-1)} \, r^{(2(i-1)-(j-1)) \otimes } \otimes I^{((j-1)-(i-1))\otimes } \right] \\&\quad = \displaystyle \sum _{i=2}^j \mu _{j-1,i-1} \nu _{i-1} \Big [ (a-2(i-1))\Vert r\Vert ^{a-2(i-1)-2} \, r^{(2(i-1)-(j-1)+1) \otimes } \otimes I^{(j-i)\otimes }\\&\qquad + ((2(i-1)-(j-1)) \Vert r\Vert ^{a-2(i-1)} \, r^{(2(i-1)-(j-1)-1)\otimes } \otimes I^{(j-1)-(i-1)+1)\otimes } \Big ]\\&\quad = \displaystyle \sum _{i=2}^j \mu _{j-1,i-1} \nu _{i-1} \Big [ (a+2-2i)\Vert r\Vert ^{a-2i} \, r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes }\\&\qquad + (2(i-1)-j+1) \Vert r\Vert ^{a-2(i-1)} \, r^{(2(i-1)-j) \otimes } \otimes I^{(j-(i-1))\otimes } \Big ]\\&\quad = \displaystyle \sum _{i=2}^j \mu _{j-1,i-1} \nu _{i-1} (a+2-2i)\Vert r\Vert ^{a-2i} \, r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes }\\&\qquad + \displaystyle \sum _{i=1}^{j-1} (2i-j+1) \mu _{j-1,i}\nu _i \Vert r\Vert ^{a-2i}\, r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes } \\&\quad = \displaystyle \sum _{i=1}^j\big ((a+2-2i)\mu _{j-1,i-1}\nu _{i-1} + (2i-j+1)\mu _{j-1,i}\nu _i \big ) \Vert r\Vert ^{a-2i} \, r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes }, \end{aligned}$$

where the last equation uses the convention that $\mu _{j,0} = 0$ and $\mu _{j-1,j} = 0$ for all j. Thus we may write

$$\begin{aligned} \nabla _\cdot ^j \Vert r\Vert ^a =\nabla _\cdot ^1\left[ \nabla _\cdot ^{j-1}\Vert r\Vert ^a \right] = \displaystyle \sum _{i=1}^j \mu _{j,i} \nu _i \, \Vert r\Vert ^{a-2i} \, r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes } \end{aligned}$$

(A.3)

with

$$\begin{aligned} \mu _{j,i}\nu _i= & {} (a+2-2i) \mu _{j-1,i-1}\nu _{i-1} + (2i-j+1) \mu _{j-1,i}\nu _i \nonumber \\= & {} \big [\mu _{j-1,i-1} + (2i-j+1) \mu _{j-1,i}\big ]\nu _i, \end{aligned}$$

(A.4)

where we used the identity

$$\begin{aligned} \nu _i = (a+2-2i)\nu _{i-1} \;\; \text{ for } \;\; i = 1, \ldots , j \end{aligned}$$

(A.5)

to deduce the second equality. Now (A.3) gives that

$$\begin{aligned} \nabla _\cdot ^j \Vert r\Vert ^a[v]^j = \displaystyle \sum _{i=1}^j \mu _{j,i} \nu _i \Vert r\Vert ^{a-j} \, \left( \frac{r^Tv}{\Vert r\Vert }\right) ^{2i-j} (v^Tv)^{j-i}. \end{aligned}$$

It is then easy to see that the maximum in (1.7) is achieved for $v = r/\Vert r\Vert$, so that

$$\begin{aligned} \Vert \, \nabla _\cdot ^j \Vert r\Vert ^a \,\Vert _{[j]} =\left| \displaystyle \sum _{i=1}^j \mu _{j,i} \nu _i \right| \Vert r\Vert ^{a-j} = |\pi _j|\, \Vert r\Vert ^{a-j} \end{aligned}$$

(A.6)

with

$$\begin{aligned} \pi _j {\mathop {=}\limits ^\mathrm{def}}\displaystyle \sum _{i=1}^{j}\mu _{j,i}\,\nu _i. \end{aligned}$$

(A.7)

Successively using this definition, (A.4), (A.5) (twice), the identity $\mu _{j-1,j} = 0$ and (A.7) again, we then deduce that

$$\begin{aligned} \pi _j= & {} \displaystyle \sum _{i=1}^{j} \mu _{j-1,i-1}\nu _i + \displaystyle \sum _{i=1}^{j} (2i-j+1) \mu _{j-1,i}\nu _i\nonumber \\= & {} \displaystyle \sum _{i=1}^{j-1} \mu _{j-1,i}\nu _{i+1} + \displaystyle \sum _{i=1}^{j} (2i-j+1) \mu _{j-1,i}\nu _i\nonumber \\= & {} \displaystyle \sum _{i=1}^{j-1} \mu _{j-1,i}\big [ \nu _{i+1} + (2i-j+1) \nu _i\big ]\nonumber \\= & {} \displaystyle \sum _{i=1}^{j-1} \mu _{j-1,i}\big [ (a+2-2(i+1))\nu _i + (2i-j+1) \nu _i\big ]\nonumber \\= & {} (a+1-j) \displaystyle \sum _{i=1}^{j-1} \mu _{j-1,i}\,\nu _i\nonumber \\= & {} (a+1-j) \pi _{j-1}. \end{aligned}$$

(A.8)

Since $\pi _1 = a$ from the first part of (A.1), we obtain from (A.8) that

$$\begin{aligned} \pi _j = \pi (a-j), \end{aligned}$$

(A.9)

which, combined with (A.6) and (A.7), gives (3.3). Moreover, (A.9), (A.7) and (A.3) give (3.2) with $\phi _{i,j}= \mu _{j,i}\,\nu _i$. In order to prove (3.4) (where now $\Vert r\Vert =1$), we use (A.3), (A.7), (A.9) and obtain that

$$\begin{aligned} \nabla _\cdot ^j \Vert \beta _1r\Vert ^a-\nabla _\cdot ^j \Vert \beta _2r\Vert ^a&= \displaystyle \sum _{i=1}^j \mu _{j,i} \nu _i \, \Vert \beta _1r\Vert ^{a-2i} \, \beta _1^{(2i-j)} r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes }\\&\quad - \displaystyle \sum _{i=1}^j \mu _{j,i} \nu _i \, \Vert \beta _2r\Vert ^{a-2i} \, \beta _2^{(2i-j)} r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes }\\&= \pi (a-j) \left[ \beta _1^{a-j}-\beta _2^{a-j}\right] \, \Vert r\Vert ^{a-2i} \, r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes }\\&= \pi (a-j) \left[ \beta _1^{a-j}-\beta _2^{a-j}\right] \, r^{(2i-j) \otimes } \otimes I^{(j-i)\otimes }. \end{aligned}$$

Using (1.7) again, it is easy to verify that the maximum defining the norm is achieved for $v=r$ and (3.4) then follows from $\Vert r\Vert =1$. $\square$