On the Convexification of Constrained Quadratic Optimization Problems with Indicator Variables

Abstract

Motivated by modern regression applications, in this paper, we study the convexification of quadratic optimization problems with indicator variables and combinatorial constraints on the indicators. Unlike most of the previous work on convexification of sparse regression problems, we simultaneously consider the nonlinear objective, indicator variables, and combinatorial constraints. We prove that for a separable quadratic objective function, the perspective reformulation is ideal independent from the constraints of the problem. In contrast, while rank-one relaxations cannot be strengthened by exploiting information from k-sparsity constraint for $k\ge 2$, they can be improved for other constraints arising in inference problems with hierarchical structure or multi-collinearity.

Keywords

Andrés Gómez is supported, in part, by grant 1930582 of the National Science Foundation. Simge Küçükyavuz is supported, in part, by ONR grant N00014-19-1-2321.

Appendix

Proof

(Theorem 2). First, note that the validity of the new inequality defining $\mathrm {cl \ conv}\left( Z_{Q_1}\right)$ follows from Proposition 1. For $a, b \in \mathbb R^{p}$ and $c \in \mathbb {R}$, consider the following two optimization problems:

$$\begin{aligned} \min \quad&a^\top z + b^\top \beta + c t \qquad ~\text{ subject } \text{ to }\;\;\;\; (z, \beta , t) \in Z_{Q_1}. \end{aligned}$$

(14)

and

$$\begin{aligned} \min \quad&a^\top z + b^\top \beta + c t \end{aligned}$$

(15a)

$$\begin{aligned} \text {subject to} \quad&(\mathbf 1^\top \beta )^2 \le t \end{aligned}$$

(15b)

$$\begin{aligned}&\frac{(\mathbf 1^\top \beta )^2}{\sum _{i \in [p]} z_i} \le t \end{aligned}$$

(15c)

$$\begin{aligned}&\sum _{i \in [p]} z_i \le k \end{aligned}$$

(15d)

$$\begin{aligned}&z \in [0,1]^p. \end{aligned}$$

(15e)

The analysis for cases where $c=0$ and $c<0$ is similar to the proof of Theorem 1, and we can proceed with assuming $c=1$ and $b \in \mathbb R^{p}$. First suppose that b is not a multiple of all-ones vector, then $\exists b_i < b_j$ for some $i,j\in [p], i\ne j$. Let $\bar{z} = e_i + e_j$, $\bar{\beta }= \tau (e_i - e_j)$ for some scalar $\tau$, and $\bar{t}=0$. Note that $(\bar{z},\bar{\beta },\bar{t})$ is feasible for both (14) and (15), and if we let $\tau$ go to infinity the objective value goes to minus infinity. So (14) and (15) are unbounded.

Now suppose that $b = \kappa \mathbf 1^\top$ for some $\kappa \in \mathbb {R}$ and $c = 1$; in this case both (14) and (15) have finite optimal value. It suffices to show that there exists an optimal solution $(z^{*}, \beta ^{*}, t^{*})$ of (15) that is integral in $z^{*}$. If $\sum _{i \in [p]} z^{*}_i = 0$, then we know $z^{*}_i =\beta _i^*= 0, \forall i \in [p]$ for both (14) and (15), and we are done. If $0< \sum _{i \in [p]} z^{*}_i < 1$ and the corresponding optimal objective value is 0 (or positive), then by letting $z^{*} =\mathbf 0$, $\beta ^{*} =\mathbf 0$ and $t^{*} = 0$, we get a feasible solution with the same objective value (or better). If $0< \sum _{i \in [p]} z^{*}_i < 1$ and $(z^{*}, \beta ^{*}, t^{*})$ attains a negative objective value, then let $\gamma = \frac{1}{\sum _{i \in [p]} z^{*}_i}$: $(\gamma z^{*}, \gamma \beta ^{*}, \gamma t^{*})$ is also a feasible solution of (15) with a strictly smaller objective value, which is a contradiction.

Finally, consider the case where $\sum _{i \in [p]} z^{*}_i \ge 1$. In this case, the constraint $(\mathbf 1^\top \beta )^2 \le t$ is active and the optimal value is attained when $\mathbf 1^\top \beta ^{*} = -\frac{\kappa }{2}$ and $t^{*} = (\mathbf 1^\top \beta ^{*})^2$, and (15) has the same optimal value as the LP:

$$\begin{aligned} \min \quad&a^\top z - \frac{\kappa ^2}{4} \qquad ~\text{ subject } \text{ to }\;\;\;\; 1 \le \sum _{i \in [p]} z_i \le k, z \in [0,1]^{p}. \end{aligned}$$

The constraint set of this LP is an interval matrix, so the LP has an integral optimal solution, $z^{*}$, hence, so does (15).    $\square$

Proof

(Lemma 1). Suppose $z^{*}$ is an extreme point of $Q_g$ and $z^{*}$ has a fractional entry. If $\sum _{i \in [p-1]} z^{*}_i - (p-2)z^{*}_p > 1$, let us consider the two cases where $z^{*}_p = 0$ and $z^{*}_p >0$. When $z^{*}_p = 0$ and there exists a fractional coordinate $z^{*}_i$ where $i \in [p-1]$, we can perturb $z^{*}_i$ by a sufficient small quantity $\epsilon$ such that $z^{*} + \epsilon e_i$ and $z^{*} - \epsilon e_i$ are in $Q_g$. Then, $z^{*} = \frac{1}{2} (z^{*} + \epsilon e_i) + \frac{1}{2} (z^{*} - \epsilon e_i)$ which contradicts the fact that $z^{*}$ is an extreme point of $Q_g$. When $1> z^{*}_p > 0$ we can perturb $z^{*}_p$ and all other $z^{*}_i$ with $z^{*}_i = z^{*}_p$ by a sufficiently small quantity $\epsilon$ and stay in $Q_g$. Similarly, we will reach a contradiction.

Now suppose $\sum _{i \in [p-1]} z^{*}_i - (p-2)z^{*}_p = 1$, and let us consider again the two cases where $z^{*}_p = 0$ and $z^{*}_p >0$. When $z^{*}_p = 0$, $z^{*} = z^{*}_1e_1 + \cdots + z^{*}_{(p-1)} e_{(p-1)}$, which is a contradiction since we can write $z^{*}$ as a convex combination of points $e_i\in Q_g, i\in [p-1]$ and there exists at least two indices $i, j \in [p-1], i\ne j$ such that $1> z^{*}_i , z^{*}_j >0$ by the fact that $z^{*}$ has a fractional entry and $\sum _{i \in [p-1]} z^{*}_i = 1, 0 \le z^{*}_i \le 1, \forall i$. When $1> z^{*}_p > 0$, we first show that there exists at most one 1 in $z^{*}_1, z^{*}_2, \dots , z^{*}_{(p-1)}$. Suppose we have $z^{*}_i =1$ and $z^{*}_j = 1$ for $i,j\in [p-1]$ with $i\ne j$, then $\sum _{i \in [p-1]} z^{*}_i - (p-2)z^{*}_p = z^{*}_i + \sum _{l \in [p-1], l \ne i} (z^{*}_{l} - z^{*}_p) \ge z^{*}_i + (z^{*}_{j} - z^{*}_p) > z^{*}_i = 1$, which is a contradiction. We now show that we can perturb $z^{*}_p$ and the $p-2$ smallest elements in $z^{*}_i, i \in [p-1]$ by a small quantity $\epsilon$ and remain in $Q_g$. The equality $\sum _{i \in [p-1]} z_i - (p-2) z_p = 1$ clearly holds after the perturbation. And, adding a small quantity $\epsilon$ to $z^{*}_p$ and the $p-2$ smallest elements in $z^{*}_i, i \in [p-1]$ will not violate the hierarchy constraint since the largest element in $z^{*}_i, i \in [p-1]$ has to be strictly greater than $z^{*}_p$. (Note that if $z^{*}_i = z^{*}_p, \forall i \in [p]$, $\sum _{i \in [p-1]} z^{*}_i - (p-2)z^{*}_p = z^{*}_p < 1$.) Since $z^{*}_i \ge z^{*}_p >0, \forall i \in [p-1]$ subtracting a small quantity $\epsilon$ will not violate the non-negativity constraint. Thus, we can write $z^{*}$ as a convex combination of two points in $Q_g$, which is a contradiction.   $\square$