Efficient differentiable quadratic programming layers: an ADMM approach

Abstract

Recent advances in neural-network architecture allow for seamless integration of convex optimization problems as differentiable layers in an end-to-end trainable neural network. Integrating medium and large scale quadratic programs into a deep neural network architecture, however, is challenging as solving quadratic programs exactly by interior-point methods has worst-case cubic complexity in the number of variables. In this paper, we present an alternative network layer architecture based on the alternating direction method of multipliers (ADMM) that is capable of scaling to moderate sized problems with 100–1000 decision variables and thousands of training examples. Backward differentiation is performed by implicit differentiation of a customized fixed-point iteration. Simulated results demonstrate the computational advantage of the ADMM layer, which for medium scale problems is approximately an order of magnitude faster than the state-of-the-art layers. Furthermore, our novel backward-pass routine is computationally efficient in comparison to the standard approach based on unrolled differentiation or implicit differentiation of the KKT optimality conditions. We conclude with examples from portfolio optimization in the integrated prediction and optimization paradigm.

A Appendix

1.1 A.1 Proof of Proposition 1

We define $\mathbf{v}^k = \mathbf{x}^{k+1} +\varvec{\mu }^k$. We can therefore express Equation (14b) as:

$$\begin{aligned} \mathbf{z}^{k+1} = \Pi ( \mathbf{x}^{k+1} + {\varvec{\mu }}^k ) = \Pi ( \mathbf{v}^k ), \end{aligned}$$

(42)

and Equation (14c) as:

$$\begin{aligned} {\varvec{\mu }}^{k+1} = {\varvec{\mu }}^{k} + \mathbf{x}^{k+1} - \mathbf{z}^{k+1} = \mathbf{v}^k - \Pi ( \mathbf{v}^k ). \end{aligned}$$

(43)

Substituting Equations (42) and (43) into Equation (14a) gives the desired fixed-point iteration:

$$\begin{aligned} \begin{bmatrix} \mathbf{v}^{k+1}\\ \varvec{\eta }^{k+2} \end{bmatrix}&= \begin{bmatrix} \mathbf{x}^{k+2} +\varvec{\mu }^{k+1}\\ \varvec{\eta }^{k+2} \end{bmatrix} \end{aligned}$$

(44)

$$\begin{aligned}&= -\begin{bmatrix} \mathbf{Q}+ \rho \mathbf{I}_{\mathbf{v}} &{} \mathbf{A}^T \\ \mathbf{A}&{} 0 \end{bmatrix}^{-1} \begin{bmatrix} \mathbf{p}- \rho (\mathbf{z}^{k+1} - \varvec{\mu }^{k+1})\\ -\mathbf{b}\end{bmatrix} + \begin{bmatrix} \varvec{\mu }^{k+1}\\ 0 \end{bmatrix} \end{aligned}$$

(45)

$$\begin{aligned}&= -\begin{bmatrix} \mathbf{Q}+ \rho \mathbf{I}_{\mathbf{v}} &{} \mathbf{A}^T \\ \mathbf{A}&{} 0 \end{bmatrix}^{-1} \begin{bmatrix} \mathbf{p}- \rho (2\Pi (\mathbf{v}^{k}) - \mathbf{v}^{k})\\ -\mathbf{b}\end{bmatrix} + \begin{bmatrix} \mathbf{v}^{k}\\ \varvec{\eta }^{k+1} \end{bmatrix} - \begin{bmatrix} \Pi (\mathbf{v}^{k})\\ \varvec{\eta }^{k+1} \end{bmatrix}. \end{aligned}$$

(46)

1.2 A.2 Proof of Proposition 2

We define $F :{\mathbb {R}}^{d_{v}} \times {\mathbb {R}}^{d_\eta } \rightarrow {\mathbb {R}}^{d_{v}} \times {\mathbb {R}}^{d_\eta }$ as:

$$\begin{aligned} F(\mathbf{v},\varvec{\eta }) = -\begin{bmatrix} \mathbf{Q}+ \rho \mathbf{I}_{\mathbf{v}} &{} \mathbf{A}^T \\ \mathbf{A}&{} 0 \end{bmatrix}^{-1} \begin{bmatrix} \mathbf{p}- \rho (2\Pi (\mathbf{v}) - \mathbf{v})\\ -\mathbf{b}\end{bmatrix} + \begin{bmatrix} \mathbf{v}\\ \varvec{\eta }\end{bmatrix} - \begin{bmatrix} \Pi (\mathbf{v})\\ \varvec{\eta }\end{bmatrix}, \end{aligned}$$

(47)

and let

$$\begin{aligned} \mathbf{M}= \begin{bmatrix} \mathbf{Q}+ \rho \mathbf{I}_{\mathbf{v}} &{} \mathbf{A}^T \\ \mathbf{A}&{} 0 \end{bmatrix}. \end{aligned}$$

(48)

Therefore we have

$$\begin{aligned} \mathbf{M}F(\mathbf{v},\varvec{\eta }) = - \begin{bmatrix} \mathbf{p}- \rho (2\Pi (\mathbf{v}) - \mathbf{v})\\ -\mathbf{b}\end{bmatrix} + \mathbf{M}\begin{bmatrix} \mathbf{v}\\ \varvec{\eta }\end{bmatrix} - \mathbf{M}\begin{bmatrix} \Pi (\mathbf{v})\\ \varvec{\eta }\end{bmatrix}. \end{aligned}$$

(49)

Taking the partial differentials of Equation (49) with respect to the relevant problem variables therefore gives:

$$\begin{aligned} \mathbf{M}\partial F(\mathbf{v},\varvec{\eta })&= - \begin{bmatrix} \partial \mathbf{p}\\ -\partial \mathbf{b}\end{bmatrix} + \partial \mathbf{M}\begin{bmatrix} \mathbf{v}\\ \varvec{\eta }\end{bmatrix} - \partial \mathbf{M}\begin{bmatrix} \Pi (\mathbf{v})\\ \varvec{\eta }\end{bmatrix} -\partial \mathbf{M}F(\mathbf{v},\varvec{\eta })\nonumber \\&= - \begin{bmatrix} \partial \mathbf{p}\\ -\partial \mathbf{b}\end{bmatrix} -\partial \mathbf{M}\Bigg [-\mathbf{M}^{-1} \begin{bmatrix} \mathbf{p}- \rho (2\Pi (\mathbf{v}) - \mathbf{v})\\ -\mathbf{b}\end{bmatrix} \Bigg ]\nonumber \\&= - \begin{bmatrix} \partial \mathbf{p}\\ -\partial \mathbf{b}\end{bmatrix} -\partial \mathbf{M}\begin{bmatrix} \mathbf{x}^* \\ \varvec{\eta }^* \end{bmatrix}\nonumber \\&= - \begin{bmatrix} \partial \mathbf{p}+ \frac{1}{2} (\partial \mathbf{Q}\mathbf{x}^* + \partial \mathbf{Q}^T\mathbf{x}^*) + \partial \mathbf{A}^T\varvec{\eta }^* \\ -\partial \mathbf{b}+ \partial \mathbf{A}\mathbf{x}^* \end{bmatrix}. \end{aligned}$$

(50)

From Equation (49) we have that the differential $\partial F(\mathbf{v},\varvec{\eta })$ is given by:

$$\begin{aligned} \partial F(\mathbf{v},\varvec{\eta }) = - \mathbf{M}^{-1} \begin{bmatrix} \partial \mathbf{p}+ \frac{1}{2} (\partial \mathbf{Q}\mathbf{x}^* + \partial \mathbf{Q}^T\mathbf{x}^*) +\partial \mathbf{A}^T\varvec{\eta }^* \\ -\partial \mathbf{b}+ \partial \mathbf{A}\mathbf{x}^* \end{bmatrix}. \end{aligned}$$

(51)

Substituting the gradient action of Equation (51) into Equation (26) and taking the left matrix-vector product of the transposed Jacobian with the previous backward-pass gradient, $\frac{\partial \ell }{\partial \mathbf{z}^*}$, gives the desired result.

$$\begin{aligned} \begin{bmatrix} \hat{ \mathbf{d}}_{\mathbf{x}} \\ \hat{ \mathbf{d}}_{\varvec{\eta }} \end{bmatrix} = \begin{bmatrix} \mathbf{Q}+ \rho \mathbf{I}_{\mathbf{v}} &{} \mathbf{A}^T \\ \mathbf{A}&{} 0 \end{bmatrix}^{-1} \Big [\mathbf{I}_{\tilde{\mathbf{v}}} - \nabla _{\tilde{\mathbf{v}}} F(\tilde{\mathbf{v}}(\varvec{\theta }),\varvec{\theta }) \Big ]^{-T} \begin{bmatrix} D\Pi (\mathbf{v}) &{} 0\\ 0 &{} \mathbf{I}_{\varvec{\eta }} \end{bmatrix} \begin{bmatrix} \big ( - \frac{\partial \ell }{\partial \mathbf{z}^*} \big )^T \\ 0 \end{bmatrix}. \end{aligned}$$

(52)

From Equation (24) we have:

$$\begin{aligned} \mathbf{I}_{\tilde{\mathbf{v}}} - \nabla _{\tilde{\mathbf{v}}} F(\tilde{\mathbf{v}}(\varvec{\theta }),\varvec{\theta }) =\begin{bmatrix} \mathbf{Q}+ \rho \mathbf{I}_{\mathbf{v}} &{} \mathbf{A}^T \\ \mathbf{A}&{} \mathbf{0}\end{bmatrix}^{-1} \begin{bmatrix} - \rho (2D\Pi (\mathbf{v}) - \mathbf{I}_{\mathbf{v}}) &{} 0\\ 0 &{} 0 \end{bmatrix} + \begin{bmatrix} D\Pi (\mathbf{v}) &{} 0\\ 0 &{} \mathbf{I}_{\varvec{\eta }} \end{bmatrix}. \end{aligned}$$

(53)

Simplifying Equation (52) with Equation (53) yields the final expression:

$$\begin{aligned} \begin{bmatrix} \hat{ \mathbf{d}}_{\mathbf{x}} \\ \hat{ \mathbf{d}}_{\varvec{\eta }} \end{bmatrix}&= \Bigg [\begin{bmatrix} D\Pi (\mathbf{v}) &{} 0\\ 0 &{} \mathbf{I}_{\varvec{\eta }} \end{bmatrix} \begin{bmatrix} \mathbf{Q}+ \rho \mathbf{I}_{\mathbf{v}} &{} \mathbf{A}^T \\ \mathbf{A}&{} 0 \end{bmatrix} + \begin{bmatrix} - \rho (2D\Pi (\mathbf{v}) - \mathbf{I}_{\mathbf{v}}) &{} 0\\ 0 &{} 0 \end{bmatrix}\Bigg ]^{-1}\nonumber \\&\quad \begin{bmatrix} D\Pi (\mathbf{v}) &{} 0\\ 0 &{} \mathbf{I}_{\varvec{\eta }} \end{bmatrix} \begin{bmatrix} \big ( - \frac{\partial \ell }{\partial \mathbf{z}^*} \big )^T \\ 0 \end{bmatrix}. \end{aligned}$$

(54)

1.3 A.3 Proof of Proposition 3

From the KKT system of equations (20) we have:

$$\begin{aligned} \mathbf{G}^T{{\,\mathrm{diag}\,}}(\tilde{\varvec{\lambda }}^*) \hat{ \mathbf{d}}_{\varvec{\lambda }} ={{\,\mathrm{diag}\,}}(\rho \varvec{\mu }^*) \hat{ \mathbf{d}}_{\varvec{\lambda }} =\Big ( - \Big (\frac{\partial \ell }{\partial \mathbf{z}^*}\Big )^T -\mathbf{Q}\hat{ \mathbf{d}}_{\mathbf{x}} - \mathbf{A}^T \hat{ \mathbf{d}}_{\varvec{\eta }} \Big ). \end{aligned}$$

(55)

From Equation (21) it follows that:

$$\begin{aligned} \frac{\partial \ell }{\partial \mathbf{l}} \ne 0 \iff \varvec{\lambda }^*_-> 0 \quad \text {and} \quad \frac{\partial \ell }{\partial \mathbf{u}} \ne 0 \iff \varvec{\lambda }^*_+ > 0, \end{aligned}$$

(56)

and therefore Equation (55) uniquely determines the relevant non-zero gradients. Let $\tilde{\varvec{\mu }}^*$ be as defined by Equation (29), then it follows that:

$$\begin{aligned} \hat{ \mathbf{d}}_{\varvec{\lambda }} = {{\,\mathrm{diag}\,}}(\rho \tilde{\varvec{\mu }}^*)^{-1} \Big ( - \Big (\frac{\partial \ell }{\partial \mathbf{z}^*}\Big )^T -\mathbf{Q}\hat{ \mathbf{d}}_{\mathbf{x}} - \mathbf{A}^T \hat{ \mathbf{d}}_{\varvec{\eta }} \Big ). \end{aligned}$$

(57)

Substituting $\hat{ \mathbf{d}}_{\varvec{\lambda }}$ into Equation (21) gives the desired gradients.

1.4 A.4 Data Summary

See Table 3.

Table 3 U.S. stock data, sorted by GICS Sector. Data provided by Quandl

1.5 A.5 Experiment 1: relative performance

See Table 4.

Table 4 Computational performance of ADMM KKT, ADMM Unroll, Optnet and SCS relative to ADMM FP for various problem sizes, $d_z$, and stopping tolerances. Batch size $=128$