Solving nearly-separable quadratic optimization problems as nonsmooth equations

Abstract

An algorithm for solving nearly-separable quadratic optimization problems (QPs) is presented. The approach is based on applying a semismooth Newton method to solve the implicit complementarity problem arising as the first-order stationarity conditions of such a QP. An important feature of the approach is that, as in dual decomposition methods, separability of the dual function of the QP can be exploited in the search direction computation. Global convergence of the method is promoted by enforcing decrease in component(s) of a Fischer–Burmeister formulation of the complementarity conditions, either via a merit function or through a filter mechanism. The results of numerical experiments when solving convex and nonconvex instances are provided to illustrate the efficacy of the method.

Appendix: Stationarity, differentiability, regularity, and (1.1) versus (1.5)

In this appendix, we discuss stationarity conditions for the QPs (1.1) and ($1.2_i$), differentiability properties of the subproblem solutions $x_i(\cdot )$ with respect to $\lambda$, regularity properties of implicit complementarity problem (1.5), and relationships between solutions of (1.1) and (1.5).

1.1 Stationarity conditions for (1.1) and ($1.2_i$)

First-order stationarity conditions for the QPs (1.1) and ($1.2_i$) can be derived from standard theory on Karush-Kuhn-Tucker conditions; e.g., see [2]. A point $\varvec{x}^*$ is a first-order KKT point of (1.1) if there exist multipliers $(\lambda ^*,\varvec{\xi }^*)$ such that

$$\begin{aligned} \begin{aligned} \varvec{Q}\varvec{x}^* + \varvec{q}+ \varvec{A}^T\lambda ^* + \varvec{B}^T\varvec{\xi }^* =&\; 0 \\ 0 \le b - \varvec{A}\varvec{x}^* \perp \lambda ^* \ge&\; 0 \\ 0 \le \varvec{c}- \varvec{B}\varvec{x}^* \perp \varvec{\xi }^* \ge&\; 0. \end{aligned} \end{aligned}$$

(6.6b)

Similarly, a point $x_i^*$ is a first-order KKT point of ($1.2_i$) if there exists multipliers $\xi _i^*$ such that

$$\begin{aligned} \begin{aligned} Q_ix_i^* + q_i + A_i^T\lambda + B_i^T\xi _i^*&= 0 \\ 0 \le c_i - B_ix_i^* \perp \xi _i^*&\ge 0. \end{aligned} \end{aligned}$$

(7.1)

Second-order KKT points are defined as those at which the first-order KKT conditions can be satisfied and at which a curvature condition on the Hessian holds over a critical cone defined by the corresponding active constraints. To formally state these second-order conditions, let us first define the following: Given a primal point $\varvec{x}\in \mathbb {R}^{nN}$, a Lagrange multiplier vector for the coupling constraint $\lambda \in \mathbb {R}^m$, and Lagrange multipliers for the subproblem constraints $\varvec{\xi }:= (\xi _1,\dots ,\xi _N) \in \mathbb {R}^{pN}$, let subsets of the indices for the coupling constraints be

$$\begin{aligned} \begin{aligned} \alpha (\varvec{x},\lambda )&:= \{ j : (\varvec{A}\varvec{x}- b)_{[j]} = 0< \lambda _{[j]} \}, \\ \beta (\varvec{x},\lambda )&:= \{ j : (\varvec{A}\varvec{x}- b)_{[j]} = 0 = \lambda _{[j]} \}, \\ \text {and}\ \ \gamma (\varvec{x},\lambda )&:= \{ j : ( \varvec{A}\varvec{x}-b)_{[j]} < 0 = \lambda _{[j]} \}, \end{aligned} \end{aligned}$$

(7.2)

and, for all $i \in \{1,\dots ,N\}$, let subsets of the indices for the inequalities in $\mathcal{X}_i$ be

$$\begin{aligned} \begin{aligned} \alpha _i(x_i,\xi _i)&:= \{ j : (B_ix_i - c_i)_{[j]} = 0< \xi _{i[j]} \}, \\ \beta _i(x_i,\xi _i)&:= \{ j : (B_ix_i - c_i)_{[j]} = 0 = \xi _{i[j]} \}, \\ \text {and}\ \ \gamma _i(x_i,\xi _i)&:= \{ j : (B_ix_i - c_i)_{[j]} < 0 = \xi _{i[j]} \}. \end{aligned} \end{aligned}$$

(7.3)

Using these definitions, given $(\varvec{x}^*,\lambda ^*,\varvec{\xi }^*)$ satisfying (7.1), we define the critical cone

$$\begin{aligned} \mathcal{T}(\varvec{x}^*,\lambda ^*,\varvec{\xi }^*) := \left\{ \varvec{d}= \begin{bmatrix} d_1 \\ \vdots \\ d_N \end{bmatrix} \in \mathbb {R}^{nN} : \left\{ \begin{aligned} \sum \limits _{i=1}^NA_{i[\alpha \cdot ]}d_i = 0,\; \sum \limits _{i=1}^NA_{i[\beta \cdot ]}d_i&\le 0, \\ \text {and}\ \ B_{i[\alpha _i\cdot ]}d_i = 0, \; B_{i[\beta _i\cdot ]}d_i&\le 0 \\ \text {for all}\ \ i \in \{1,\dots ,N\} \end{aligned} \right\} \right\} . \end{aligned}$$

(7.4)

Similarly, corresponding to a point $(x_i^*,\xi _i^*)$ satisfying (7.2), we define the critical cone

$$\begin{aligned} {\mathcal {T}}_i(x_i^*,\xi _i^*) := \left\{ d_i \in \mathbb {R}^n : B_{i[\alpha _i\cdot ]}d_i = 0\ \ \text {and}\ \ B_{i[\beta _i\cdot ]}d_i \le 0 \right\} , \end{aligned}$$

(7.5)

We may now state that $\varvec{x}^*$ is a second-order KKT point of (1.1) if there exist multipliers $(\lambda ^*,\varvec{\xi }^*)$ such that (7.1) holds and

$$\begin{aligned} \varvec{d}^T\varvec{Q}\varvec{d}\ge 0\ \ \text {for all}\ \ \varvec{d}\in \mathcal{T}(\varvec{x}^*,\varvec{\xi }^*,\lambda ^*), \end{aligned}$$

(7.6)

and $x_i^*$ is a second-order KKT point of ($1.2_i$) if there exists $\xi _i^*$ such that (7.2) holds and

$$\begin{aligned} d_i^TQ_id_i \ge 0\ \ \text {for all}\ \ d_i \in \mathcal{T}_i(x_i^*,\xi _i^*). \end{aligned}$$

(7.7)

If these curvature conditions hold strictly for nonzero elements in the critical cone, then one can conclude that the corresponding point is a strict local minimizer. In particular, $\varvec{x}^*$ is a strict local minimizer of (1.1) if there exist multipliers $(\lambda ^*,\varvec{\xi }^*)$ such that (7.1) holds and

$$\begin{aligned} \varvec{d}^T\varvec{Q}\varvec{d}> 0\ \ \text {for all}\ \ \varvec{d}\in \mathcal{T}(\varvec{x}^*,\varvec{\xi }^*,\lambda ^*) \setminus \{0\}, \end{aligned}$$

(7.8)

while $x_i^*$ is a strict local minimizer of ($1.2_i$) if there exists $\xi _i^*$ such that (7.2) holds and

$$\begin{aligned} d_i^TQ_id_i > 0\ \ \text {for all}\ \ d_i \in \mathcal{T}_i(x_i^*,\xi _i^*) \setminus \{0\}. \end{aligned}$$

(7.9)

Overall, one can observe that if (1.1) (resp. ($1.2_i$)) is convex in that $\varvec{Q}\succeq 0$ (resp. $Q_i \succeq 0$), then any first-order KKT point is a second-order KKT point, from which convexity implies that it is a global minimizer of (1.1) (resp. ($1.2_i$)). Similarly, if (1.1) (resp. ($1.2_i$)) is strictly convex in that $\varvec{Q}\succ 0$ (resp. $Q_i \succ 0$), then any first-order KKT point is a strict local minimizer, from which convexity implies that it is a strict global minimizer of (1.1) (resp. ($1.2_i$)).

1.2 Differentiability concepts of locally Lipschitz functions

Let ${\varPhi } : \mathbb {R}^m \rightarrow \mathbb {R}^m$ be locally Lipschitz. Then, by Rademacher’s theorem, ${\varPhi }$ is differentiable almost everywhere in $\mathbb {R}^m$. If $D_{\varPhi }$ denotes the set of $\lambda \in \mathbb {R}^m$ at which ${\varPhi }$ is differentiable, then the B-subdifferential of ${\varPhi }$ at $\lambda$ can be defined (see [25]) as

$$\begin{aligned} \partial _B{\varPhi }(\lambda ) := \left\{ H \in \mathbb {R}^{m\times m} : H = \lim \limits _{\lambda ^k \in D_{\varPhi },\lambda ^k \rightarrow \lambda } \nabla {\varPhi }(\lambda ^k) \right\} . \end{aligned}$$

(7.10)

The convex hull of this set is the generalized Jacobian of ${\varPhi }$ at $\lambda$ [25], written as

$$\begin{aligned} \partial {\varPhi } := \text {conv}\;\partial _B{\varPhi }(\lambda ). \end{aligned}$$

Such a function is semismooth [25, 41] at $\lambda$ if it is directionally differentiable at $\lambda$ and

$$\begin{aligned} H\delta - {\varPhi }'(\lambda ;\delta ) = o(\Vert \delta \Vert )\ \ \text {for any}\ \ \delta \rightarrow 0\ \ \text {and}\ \ H \in \partial {\varPhi }(\lambda +\delta ). \end{aligned}$$

Further, ${\varPhi }$ is strongly semismooth at $\lambda$ if

$$\begin{aligned} H\delta - {\varPhi }'(\lambda ;\delta ) = o(\Vert \delta \Vert ^2)\ \ \text {for any}\ \ \delta \rightarrow 0\ \ \text {and}\ \ H \in \partial {\varPhi }(\lambda +\delta ). \end{aligned}$$

1.3 Regularity properties of solutions of (1.5)

A square matrix $M\in \mathbb {R}^{m\times m}$ is said to be a P-matrix if all of its principal minors are positive [42, Def. 3.3.1]. If $M \in \mathbb {R}^{m \times m}$ is a symmetric P-matrix, then M is positive definite. This property is of importance for matrices arising in the following definition, which relates to regularity of solutions for (1.5); see [47, Def. 2.1].

Definition 7.1

(b- and R-regular solutions of (1.5)) Suppose that $F \in C^1$. Let $\lambda ^*$ be a solution of (1.5) and let $\alpha$, $\beta$, and $\gamma$ be defined as in (7.3). Then, $\lambda ^*$ is a

• b-regular solution of (1.5) if $\nabla F_{[\sigma \sigma ]}(\lambda ^*)$ is nonsingular whenever $\alpha \subseteq \sigma \subseteq \alpha \cup \beta$;

• R-regular solution of (1.5) if $\nabla F_{[\alpha \alpha ]}(\lambda ^*)$ is nonsingular and its Schur complement w.r.t.

  $$\begin{aligned} \begin{bmatrix} \nabla F_{[\alpha \alpha ]}(\lambda ^*)&\nabla F_{[\alpha \beta ]}(\lambda ^*) \\ \nabla F_{[\beta \alpha ]}(\lambda ^*)&\nabla F_{[\beta \beta ]}(\lambda ^*) \end{bmatrix}, \end{aligned}$$

  namely

  $$\begin{aligned} \begin{aligned} (\nabla F_{[\beta \beta ]}/\nabla F_{[\alpha \alpha ]})(\lambda ^*) := \nabla F_{[\beta \beta ]}(\lambda ^*) - \nabla F_{[\beta \alpha ]}(\lambda ^*) (\nabla F_{[\alpha \alpha ]}(\lambda ^*))^{-1} \nabla F_{[\alpha \beta ]}(\lambda ^*) \end{aligned} \end{aligned}$$

  (7.11)

  is a P-matrix.

1.4 Differentiability properties of $x_i(\lambda )$

The differentiability properties of a first-order KKT point $x_i^*(\lambda )$ of ($1.2_i$) and its corresponding objective value as functions of $\lambda$ follow from the theory of sensitivity analysis of parametric nonlinear optimization problems. The study of solution and objective sensitivity to variations in parameters in such problems has been studied extensively since the 1970’s. Since the constraints of (1.1) and ($1.2_i$) are affine, we have that the conditions in Appendix 1 are necessary for stationarity with the last conditions sufficient for (local) optimality. However, for certain results below, we rely on additional regularity conditions for ($1.2_i$).

We begin by stating two types of regularity conditions on the constraints in ($1.2_i$).

Definition 7.2

(($1.2_i$)-LICQ) If $B_{i[(\alpha _i\cup \beta _i)\cdot ]}$ has full row rank (with $(\alpha _i,\beta _i)$ from (7.4)), then the Linear Independence Constraint Qualification (LICQ) for ($1.2_i$) holds at $x_i \in \mathcal{X}_i$.

Definition 7.3

(($1.2_i$)-SCQ) If there exists $x_i \in \mathbb {R}^n$ such that $B_i x_i < c_i$, then the Slater Constraint Qualification (SCQ) holds for ($1.2_i$).

If a problem has a convex feasible region, as does ($1.2_i$), the SCQ is equivalent to the Mangasarian-Fromovitz Constraint Qualification (MFCQ) holding at every feasible point. If the LICQ holds at any feasible point, then the SCQ holds. For a local minimizer $x_i^*(\lambda )$ of ($1.2_i$), the LICQ implies uniqueness of the optimal Lagrange multiplier vector $\xi ^*_i(\lambda )$; however, the SCQ only implies that the set of optimal multipliers at $x_i^*(\lambda )$, call it $\Xi _i(x_i^*(\lambda ))$, is bounded.

Now, given a first-order KKT point of ($1.2_i$) and the critical cone

$$\begin{aligned} \mathcal{T}^{sc}_i(x_i^*,\xi _i^*) := \{ d_i \in \mathbb {R}^n : B_{i[\alpha _i\cdot ]}d_i = 0\} \supseteq \mathcal{T}_i(x_i^*,\xi _i^*) \end{aligned}$$

(7.12)

with $\alpha _i$ defined in (7.4), we define the following.

Definition 7.4

(($1.2_i$)-SSOSC) If $(x_i^*,\xi _i^*)$ is a first-order KKT point of ($1.2_i$), then the strong second-order sufficient condition (SSOSC) for ($1.2_i$) holds at $(x_i^*,\xi _i^*)$ if

$$\begin{aligned} d_i^TQ_id_i > 0\ \ \text {for all}\ \ d_i \in \mathcal{T}^{sc}_i(x_i^*,\xi _i^*). \end{aligned}$$

(7.13)

An early result on differentiability of $x_i(\cdot )$ for our purposes can be derived from the work of Fiacco [13, Thm. 2.1]. We state it as the following lemma.

Lemma 7.1

Given $\bar{\lambda } \in \mathbb {R}^m$, suppose that $x_i^*(\bar{\lambda })$ is a strict local solution of ($1.2_i$) with $\lambda =\bar{\lambda }$ such that with the corresponding Lagrange multiplier $\xi _i^*(\bar{\lambda })$ the following hold:

• $\beta _i(x_i^*(\bar{\lambda }),\xi _i^*(\bar{\lambda })) = \emptyset$,

• the ($1.2_i$)-LICQ holds at $x_i^*(\bar{\lambda })$, and

• the ($1.2_i$)-SSOSC holds at $(x_i^*(\bar{\lambda }),\xi _i^*(\bar{\lambda }))$.

Then, there exist open neighborhoods $U_i$, $V_i$, and $W_i$ centered at $x_i^*$, $\xi _i^*$, and $\bar{\lambda }$, respectively, and functions $x_i(\cdot ) : W_i \rightarrow U_i \subseteq \mathcal{X}_i$ and $\xi _i(\cdot ) : W_i \rightarrow V_i$ such that:

• (i) $x_i(\cdot )$ and $\xi _i(\cdot )$ are $C^1$ functions of $\lambda \in W_i$;

• (ii) for each $\lambda \in W_i$, the point $x_i(\lambda )$ is a strict local solution of ($1.2_i$) with Lagrange multiplier $\xi _i(\lambda )$; and

• (iii) for each $\lambda \in W_i$, the Jacobian of $(x_i(\cdot ),\xi _i(\cdot ))$ at $\lambda$ is given by

  $$\begin{aligned} \begin{bmatrix} Q_i&B^T_{i[\alpha _i\cdot ]} \\ B_{i[\alpha _i\cdot ]}&0 \end{bmatrix} \begin{bmatrix} \nabla x_i(\lambda ) \\ \nabla \xi _{i[\alpha _i]}(\lambda ) \end{bmatrix} = \begin{bmatrix} -A_i^T \\ 0 \end{bmatrix}. \end{aligned}$$

  (7.14)

The following result, due to Ralph and Dempe [6, Thms. 1 & 2 and Cor. 4(2)], represents an extension of this result in which the strict complementarity requirement (i.e., $\beta _i = \emptyset$) is relaxed and the LICQ is replaced by the (weaker) MFCQ along with the Constant Rank Constraint Qualification (CRCQ). The CRCQ holds automatically for (1.1), so we do not state it explicitly. Moreover, the MFCQ is equivalent to the SCQ for (1.1), so we refer to the latter.

Lemma 7.2

Given $\bar{\lambda } \in \mathbb {R}^m$, suppose that $x_i^*(\bar{\lambda })$ is a strict local solution of ($1.2_i$) with $\lambda =\bar{\lambda }$ such that the following hold:

• the ($1.2_i$)-SCQ holds and

• the ($1.2_i$)-SSOSC holds for all $\xi _i \in \Xi _i(x_i^*(\bar{\lambda }))$.

Then, for all $i \in \{1,\dots ,N\}$, there exist open neighborhoods $U_i$ and $W_i$ centered at $x_i^*(\bar{\lambda })$ and $\bar{\lambda }$, respectively, and a function $x_i(\cdot ) : W_i \rightarrow U_i \subseteq \mathcal{X}_i$ such that:

• (i) $x_i(\cdot )$ is a $PC^1$ function of $\lambda \in W_i$;

• (ii) for each $\lambda \in W_i$, the point $x_i(\lambda )$ is a strict local solution of ($1.2_i$); and

• (iii) the directional derivative $x'_i(\lambda ;\cdot )$ is a piecewise linear function such that for each $\lambda \in W_i$, $d \in \mathbb {R}^m$, and $\xi _i \in \Xi _i(x_i(\lambda ))$ the value $x_i'(\lambda ;\delta )$ is given by the unique solution of

  $$\begin{aligned} \min \limits _v\; \tfrac{1}{2}v^TQ_iv + v^TA_i^T\delta \ \ \text {s.t.}\ \ B_{i[\alpha _i\cdot ]}v = 0,\ \ B_{i[\beta _i\cdot ]}v \le 0. \end{aligned}$$

  (7.15)

We remark that the conclusion of Lemma 7.2, namely, that $x_i(\cdot )$ is a $PC^1$ function of $\lambda \in W_i$, implies that $x_i(\cdot )$ is locally Lipschitz and directionally differentiable.

1.5 Relationship between the QP and the ICP formulations

A solution to (1.5) easily yields a first-order KKT point of (1.1). In particular, we have the following result whose proof follows simply by combining (7.2) with (1.5).

Lemma 7.3

Suppose $\lambda ^*$ solves (1.5) with $x_i^*$ denoting a first-order KKT point of ($1.2_i$) for $\lambda = \lambda ^*$ for all $i \in \{1,\dots ,N\}$. Then, $\varvec{x}^*$ is a first-order KKT point of (1.1).

Similarly, a solution to (1.5) can also yield a second-order KKT point of (1.1). For this claim, we have the following result whose proof follows by combining (7.8) and the fact that the Cartesian product of the $\mathcal{T}_i$’s represents a superset of $\mathcal{T}$.

Lemma 7.4

Suppose $\lambda ^*$ solves (1.5) with $x_i^*$ denoting a second-order KKT point of ($1.2_i$) for $\lambda = \lambda ^*$ for all $i \in \{1,\dots ,N\}$. Then, $\varvec{x}^*$ is a second-order KKT point of (1.1).

The reverse implications are the subjects of the following lemmas, the proofs of which are straightforward; hence, we state them without proof.

Lemma 7.5

Suppose that $\varvec{x}^*$ is a first-order KKT point of (1.1) in that there exist multipliers $(\lambda ^*,\varvec{\xi }^*)$ such that (7.1) holds. Then, the following hold:

• (i) for each $i \in \{1,\dots ,N\}$, $x_i^*$ is a first-order KKT point for ($1.2_i$) for $\lambda = \lambda ^*$, and

• (ii) $\lambda ^*$ solves (1.5).

Lemma 7.6

Suppose that $\varvec{x}^*$ is a second-order KKT point of (1.1) in that there exist multipliers $(\varvec{\xi }^*,\lambda ^*)$ such that (7.1) and (7.7) hold. Moreover, suppose that

$$\begin{aligned} \varvec{d}^T\varvec{Q}\varvec{d}\ge 0\ \ \text {for all}\ \ \varvec{d}= (d_1,\ldots ,d_N) \in \prod \limits _{i=1}^N\mathcal{T}_i(x_i^*,\xi _i^*). \end{aligned}$$

(7.16)

Then, the following hold:

• (i) for each $i \in \{1,\dots ,N\}$, $x_i^*$ is a second-order KKT point for ($1.2_i$) for $\lambda = \lambda ^*$, and

• (ii) $\lambda ^*$ solves (1.5).