Solution refinement at regular points of conic problems

Abstract

Many numerical methods for conic problems use the homogenous primal–dual embedding, which yields a primal–dual solution or a certificate establishing primal or dual infeasibility. Following Themelis and Patrinos (IEEE Trans Autom Control, 2019), we express the embedding as the problem of finding a zero of a mapping containing a skew-symmetric linear function and projections onto cones and their duals. We focus on the special case when this mapping is regular, i.e., differentiable with nonsingular derivative matrix, at a solution point. While this is not always the case, it is a very common occurrence in practice. In this paper we do not aim for new theorerical results. We rather propose a simple method that uses LSQR, a variant of conjugate gradients for least squares problems, and the derivative of the residual mapping to refine an approximate solution, i.e., to increase its accuracy. LSQR is a matrix-free method, i.e., requires only the evaluation of the derivative mapping and its adjoint, and so avoids forming or storing large matrices, which makes it efficient even for cone problems in which the data matrices are given and dense, and also allows the method to extend to cone programs in which the data are given as abstract linear operators. Numerical examples show that the method improves an approximate solution of a conic program, and often dramatically, at a computational cost that is typically small compared to the cost of obtaining the original approximate solution. For completeness we describe methods for computing the derivative of the projection onto the cones commonly used in practice: nonnegative, second-order, semidefinite, and exponential cones. The paper is accompanied by an open source implementation.

Appendices

Appendix A

Differentiability properties of the residual map Let C be a nonempty closed convex subset of ${\mathbf{R}}^n$. It is well known that the projection $\Pi _C$ onto C is (firmly) nonexpansive (see, e.g., [5, Proposition 2]), hence it is Lipschitz continuous with a Lipschitz constant at most 1. Consequently, if $A: {\mathbf{R}}^n\rightarrow {\mathbf{R}}^m$ is linear then the composition $A\circ \Pi _C$ is also Lipschitz continuous. Therefore, by the Rademacher Theorem (see, e.g., [37, Theorem 9.60] or [12, Theorem 3.2]) both $\Pi _C$ and $A\circ \Pi _C$ are differentiable almost everywhere. This allows us to conclude that the residual map (7) is differentiable almost everywhere. Moreover, let $z\in {\mathbf{R}}^{m+n+1}$. Clearly ${\mathcal {R}}$ is differentiable at z if $\Pi$ is differentiable at z.

Appendix B

Semi-definite cone projection derivative

Let $X\in {\mathbf{S}}^n$, let $X=U\mathbf {diag} (\lambda ) U^T$ be an eigendecomposition of X and suppose that $\det (X)\ne 0$. Without loss of generality, we can and do assume that the entries of $\lambda$ are in an increasing order. That is, there exists $k\in \{1,\ldots , n\}$ such that

$$\begin{aligned} \lambda _1\le \cdots \le \lambda _k<0<\lambda _{k+1} \le \cdots \le \lambda _n. \end{aligned}$$

(17)

We also note that

$$\begin{aligned} \Pi X - X = U\mathbf {diag} (\lambda _- ) U^T, \end{aligned}$$

(18)

where $\lambda _- = -\min (\lambda , 0)$. It follows from (11), (18), and the orthogonality of U that

$$\begin{aligned} U^T\Pi X U = \mathbf {diag} (\lambda _+ ), \quad U^T(\Pi X -X)U= \mathbf {diag} (\lambda _-). \end{aligned}$$

(19)

Note that

$$\begin{aligned} \Pi X (\Pi X - X) =U \mathbf {diag}(\lambda _+ ) \mathbf {diag} (\lambda _- ) U^T= 0. \end{aligned}$$

(20)

Let ${\mathsf {D}} {\Pi }(X): {\mathbf{S}}^n \rightarrow {\mathbf{S}}^n$ be the derivative of $\Pi$ at X, and let ${\widetilde{X}}\in {\mathbf{S}}^n$. We now show that (12) holds.

Indeed, using the first order Taylor approximation of $\Pi$ around X, for $\Delta X\in {\mathbf{S}}^n$ such that $||\Delta X||_F$ is sufficiently small (here $||\cdot ||_F$ denotes the Frobenius norm) we have

$$\begin{aligned} \Pi (X + \Delta X) \approx \Pi X + {\mathsf {D}} \Pi (X)(\Delta X). \end{aligned}$$

(21)

To simplify the notation, we set $\Delta Y={\mathsf {D}} \Pi (X)(\Delta X)$. Now

$$\begin{aligned} 0&=\Pi (X + \Delta X) (\Pi (X + \Delta X) - X - \Delta X) \end{aligned}$$

(22a)

$$\begin{aligned}&\approx (\Pi X + \Delta Y) (\Pi X + \Delta Y - X - \Delta X) \end{aligned}$$

(22b)

$$\begin{aligned}&=\Pi X(\Pi X-X) +\Delta Y(\Pi X-X) +\Pi X(\Delta Y-\Delta X) +\Delta Y (\Delta Y-\Delta X) \nonumber \\&\approx \Pi X(\Delta Y-\Delta X) + \Delta Y(\Pi X-X) \end{aligned}$$

(22c)

$$\begin{aligned}&\approx U^T\Pi X(\Delta Y-\Delta X)U + U^T\Delta Y(\Pi X-X)U \end{aligned}$$

(22d)

$$\begin{aligned}&=(U^T\Pi XU) U^T(\Delta Y-\Delta X)U + U^T\Delta YU (U^T(\Pi X-X)U) \end{aligned}$$

(22e)

$$\begin{aligned}&=\mathbf {diag} (\lambda _+ ) U^T(\Delta Y-\Delta X)U + U^T\Delta YU (\mathbf {diag} (\lambda _- )). \end{aligned}$$

(22f)

Here, (22a) follows from applying (20) with X replaced by $X + \Delta X$, (22b) follows from combining (22a) and (21), (22c) follows from (20) by neglecting second order terms, (22d) follows from multiplying (22c) from the left by $U^T$ and from the right by U, (22e) follows from the fact that $UU^T=I$ and finally (22f) follows from (19). We rewrite the Sylvester [17, 42] Eq. (22f) as

$$\begin{aligned} \mathbf {diag} (\lambda _+ ) U^T\Delta YU + U^T\Delta YU \mathbf {diag} (\lambda _-) \approx \mathbf {diag} (\lambda _+ ) U^T\Delta X U. \end{aligned}$$

(23)

Using (23), we learn that for any $i \in \{1, \ldots , n\}$ and $j \in \{1, \ldots , n\}$, we have

$$\begin{aligned} ((\lambda _-)_j +(\lambda _+)_i)(U^T\Delta Y U)_{ij} \approx (\lambda _+)_i(U^T\Delta X U)_{ij} . \end{aligned}$$

Recalling (17), if $i \le k, \, j > k$ we have $(\lambda _-)_j = (\lambda _+)_i=0$. Otherwise, $(\lambda _-)_j +(\lambda _+)_i\ne 0$ and

$$\begin{aligned} (U^T\Delta Y U)_{ij} \approx \underbrace{\frac{(\lambda _+)_i}{(\lambda _-)_j +(\lambda _+)_i}}_{=B_{ij}} (U^T\Delta X U)_{ij} . \end{aligned}$$

(24)

Proceeding by cases in view of (17), and using that $\Delta Y$ is symmetric (so is $U^T\Delta Y U$), we conclude that

$$\begin{aligned} B_{ij} = {\left\{ \begin{array}{ll} 0, &{}~~\text {if}~~i \le k, \, j \le k; \\ \frac{(\lambda _+)_i}{(\lambda _-)_j+(\lambda _+)_i}, &{} ~~\text {if}~~i> k, \, j \le k; \\ \frac{(\lambda _+)_j}{(\lambda _-)_i+(\lambda _+)_j}, &{}~~\text {if}~~i \le k, \, j> k; \\ 1,&{}~~\text {if}~~i> k, \, j > k.\\ \end{array}\right. } \end{aligned}$$

Therefore, combining with (24) we obtain

$$\begin{aligned} U^T\Delta Y U \approx B \circ (U^T\Delta X U ), \end{aligned}$$

where “$\circ$” denotes the Hadamard (i.e., entrywise) product. Recalling the definition of $\Delta Y$ and using that $UU^T=I$ we conclude that

$$\begin{aligned} {\mathsf {D}} \Pi (X)(\Delta X) \approx U (B \circ (U^T\Delta X U )) U^T. \end{aligned}$$

Letting $||\Delta X||_F\rightarrow 0$ and applying the implicit function theorem, we conclude that (12) holds.

Appendix C

Exponential cone projection derivative The Lagrangian of the constrained optimization problem (13) is

$$\begin{aligned} \tfrac{1}{2}||(x,y,z) - ({\overline{x}},{\overline{y}},{\overline{z}})||^2 +\mu ({\overline{y}} e^{{\overline{x}}/{\overline{y}}}-{\overline{z}}), \end{aligned}$$

where $\mu \in {\mathbf{R}}$ is the dual variable. The KKT conditions at a solution $(x^*,y^*,z^*,\mu ^*)$ are

$$\begin{aligned} x^*-x+\mu ^*e^{x^*/y^*}&=0\nonumber \\ y^*-y+\mu ^*e^{x^*/y^*}\big (1-\tfrac{x^*}{y^*}\big )&=0\nonumber \\ z^*-z-\mu ^*&=0\nonumber \\ y^*e^{x^*/y^*}-z^*&=0. \end{aligned}$$

(25)

Considering the differentials dx, dy, dz and $dx^*,dy^*,dz^*, d\mu ^*$ of the KKT conditions in (25), the authors of [1, Lemma 3.6] obtain the system of equations

$$\begin{aligned} \underbrace{ \begin{bmatrix} 1+\tfrac{\mu ^*e^{x^*/y^*}}{y^*}&-\tfrac{\mu ^*x^*e^{x^*/y^*}}{{y^*}^2}&0&e^{x^*/y^*} \\ -\tfrac{\mu ^*x^*e^{x^*/y^*}}{{y^*}^2}&1+\tfrac{\mu ^*{x^*}^2e^{x^*/y^*}}{{y^*}^3}&0&(1-x^*/y^*)e^{x^*/y^*} \\ 0&0&1&-1 \\ e^{x^*/y^*}&(1-x^*/y^*)e^{x^*/y^*}&-1&0 \end{bmatrix} }_{D} \underbrace{ \begin{bmatrix} dx^*\\ dy^*\\ dz^*\\ d\mu ^* \end{bmatrix} }_{du^*} =\underbrace{\begin{bmatrix} dx\\ dy\\ dz\\ 0 \end{bmatrix} }_{du}.\nonumber \\ \end{aligned}$$

(26)

Note that, since (13) is feasible, D is invertible. Therefore, $du^*=D^{-1}(du)$. Consequently, the upper left $3\times 3$ block matrix of $D^{-1}$ is the Jacobian of the projection at (x, y, z) in Case 4.