Radial duality part II: applications and algorithms

Abstract

The first part of this work established the foundations of a radial duality between nonnegative optimization problems, inspired by the work of Renegar (SIAM J Optim 26(4): 2649-2676, 2016). Here we utilize our radial duality theory to design and analyze projection-free optimization algorithms that operate by solving a radially dual problem. In particular, we consider radial subgradient, smoothing, and accelerated methods that are capable of solving a range of constrained convex and nonconvex optimization problems and that can scale-up more efficiently than their classic counterparts. These algorithms enjoy the same benefits as their predecessors, avoiding Lipschitz continuity assumptions and costly orthogonal projections, in our newfound, broader context. Our radial duality further allows us to understand the effects and benefits of smoothness and growth conditions on the radial dual and consequently on our radial algorithms.

Appendices

Appendix A: LogSumExp gradients and QP optimality certificates

In our quadratic programming example (8) and our generalized setting (37), we consider smoothings of a finite maximum. Given a smooth convex functions $f_i:\mathcal {E}\rightarrow \mathbb {R}$, we considered the smoothing of $\max \{f_i\}$ with parameter $\eta >0$ given by $f_\eta (x):= \eta \log \left( \sum _{i=0}^n \exp (f_i(x)/\eta )\right) .$ Its gradient is given by $\nabla f_\eta (x) = \sum \lambda _i \nabla f_i(x)$ where $\lambda _i = \exp (f_i(x)/\eta ) / \sum _j \exp (f_j(x)/\eta )$. Computationally evaluating this requires mild care to avoid precision issues with exponentiating potentially larger numbers. It is numerically stable to instead compute these coefficients via the equivalent formula

$$\begin{aligned} \lambda _i = \frac{\exp ((f_i(x) - \max \{f_k(x)\})/\eta )}{\sum _j \exp ((f_j(x) - \max \{f_k(x)\})/\eta )} \ . \end{aligned}$$

Next, we specialize this formula to the setting of quadratic programming for $g_\eta$ in (8). Observe the gradient of the objective component is given by

$$\begin{aligned} \nabla \left( \frac{c^Ty+1 + \sqrt{(c^Ty+1)^2 +2y^TQy}}{2}\right) _+\ (y)= \frac{Qx +c}{1+\frac{1}{2}x^TQx} \end{aligned}$$

where $x=y/\left( \frac{c^Ty+1 + \sqrt{(c^Ty+1)^2 +2y^TQy}}{2}\right) _+$ by using the gradient formula (25). The gradients of the transformed constraints are simply $\nabla a_i^T y /b_i = a_i/b_i$. Then the gradient of the smoothing overall is given by

$$\begin{aligned} \nabla g_\eta (y) = \lambda _0 \frac{Qx +c}{1+\frac{1}{2}x^TQx} + \sum _{i=1}^n \lambda _i a_i/b_i \ . \end{aligned}$$

This gradient can be computed using two matrix multiplications with A: Ay is needed to compute the coefficients $\lambda _i$, then $A^T[\lambda _1/b_1 \dots \lambda _n/b_n]$ is needed for the summation above. This gradient formula indicates a reasonable selection of dual multipliers $v_i = \frac{\lambda _i (1+\frac{1}{2}x^TQx)}{\lambda _0b_i}$ as we then have $g_\eta (y)$ proportional to $Qx+c + A^Tv$.

Appendix B: Calculation of nonconvex subgradient method guarantee (41)

Here we derive (41) from the convergence theory of [10, Theorem 3.1], which assumes the function being minimized g is uniformly M-Lipschitz and $\rho$-weakly convex (defined as $g+\frac{\rho }{2}\Vert \cdot \Vert ^2$ being convex). Equation [10, (3.4)] ensures the subgradient method $y_{k+1} = y_k - \alpha \zeta _k$ for $\zeta _k\in \partial _P g(y_k)$ has some $y_k$ with

$$\begin{aligned} \Vert \nabla g_{1/\bar{\rho }}(y_k)\Vert ^2 \le \frac{\bar{\rho }}{\bar{\rho }- \rho } \frac{(g(x_0) - \inf g) + \frac{\bar{\rho }M^2}{2}\sum _{t=0}^T\alpha _t }{\sum _{t=0}^T \alpha _t} \end{aligned}$$

where $g_{1/\bar{\rho }}$ is the Moreau envelope of g (and using that $g_{1/\bar{\rho }}(x_0) < g(x_0)$). Given the method will be run for T steps, setting $\bar{\rho }= 2\rho$ and $\alpha _k = \sqrt{\frac{g(y_0)-\inf g}{\rho M^2(T+1)}}$ this bound becomes

$$\begin{aligned} \Vert \nabla g_{1/2\rho }(y_k)\Vert ^2 \le 4\sqrt{\frac{\rho M^2(g(y_0)-\inf g)}{T+1}} \end{aligned}$$

Following the discussion of [10, Page 210], this implies $y_k$ has a nearby y that is nearly stationary on g. In particular, $\Vert y-y_k\Vert \le (1/2\rho )\Vert \nabla g_{1/2\rho }(y_k)\Vert$ and $\textrm{dist}(0,\partial _P g(y))\le \Vert \nabla g_{1/2\rho }(y_k)\Vert$. Combining this with the above bound, we conclude