The space decomposition theory for a class of eigenvalue optimizations

Abstract

In this paper we study optimization problems involving eigenvalues of symmetric matrices. One of the difficulties with numerical analysis of such problems is that the eigenvalues, considered as functions of a symmetric matrix, are not differentiable at those points where they coalesce. Here we apply the $\mathcal{U}$-Lagrangian theory to a class of D.C. functions (the difference of two convex functions): the arbitrary eigenvalue function λ _i , with affine matrix-valued mappings, where λ _i is a D.C. function. We give the first-and second-order derivatives of ${\mathcal{U}}$-Lagrangian in the space of decision variables R ^m when transversality condition holds. Moreover, an algorithm framework with quadratic convergence is presented. Finally, we present an application: low rank matrix optimization; meanwhile, list its $\mathcal{VU}$ decomposition results.

Appendix

Next we will solve the (4.5) of Proposition 4.1.

$$\begin{aligned} \mathcal{A}^{\ast} \bigl( Q_1 \bigl(A(\hat{x}) \bigr) Z Q^{T}_1 \bigl(A(\hat{x})\bigr) \bigr) = & \begin{bmatrix} \langle Q_1 (A(\hat{x})) Z Q^{T}_1 (A(\hat{x})), A_1\rangle\\ \vdots\\ \langle Q_1 (A(\hat{x})) Z Q^{T}_1 (A(\hat{x})), A_n\rangle\\ \end{bmatrix} \\ =& \begin{bmatrix} \langle Z, Q^{T}_1 (A(\hat{x})) A_1 Q_1 (A(\hat{x})) \rangle\\ \vdots\\ \langle Z, Q^{T}_1 (A(\hat{x})) A_n Q_1 (A(\hat{x})) \rangle\\ \end{bmatrix} \end{aligned}$$

We denote $W_{i} = Q^{T}_{1} (A(\hat{x})) A_{i} Q_{1} (A(\hat{x})), i=1,\ldots,n$, so the above formula can be converted to

$$ \mathcal{A}^{\ast} \bigl( Q_1 \bigl(A(\hat{x})\bigr) Z Q^{T}_1 \bigl(A(\hat{x})\bigr) \bigr) = \begin{bmatrix} \langle Z, W_1\rangle\\ \vdots\\ \langle Z, W_n\rangle\\ \end{bmatrix} . $$

The objective function $= \sum^{n}_{i=1} {\langle Z, W_{i}\rangle }^{2}$. Hence the problem becomes

$$\begin{aligned} \min &\quad \sum^n_{i=1} { \langle Z, W_i\rangle}^2 \\ \textrm{s.t.} &\quad \langle Z, I_{q-p}\rangle=1. \end{aligned}$$

According to related content about matrix analysis, S _n is isomorphic to $\mathbb{R}^{\frac{n(n+1)}{2}}$ via the map svec(A) defined by stacking the columns of the lower triangle of A on top of each other and multiplying the off-diagonal elements with $\sqrt{2}$,

$$ \textrm{svec}(A):=[a_{11},\sqrt{2}a_{21},\ldots,\sqrt {2}a_{n1}, a_{22},\sqrt{2}a_{32}, \ldots,a_{nn}]^T. $$

The factor $\sqrt{2}$ for off-diagonal elements ensures that, for A,B∈S _n ,

$$ \langle A, B\rangle= \textrm{tr}(AB) =\textrm{svec}(A)^T \textrm{svec}(B). $$

So we have 〈Z,W _i 〉=svec(Z)^Tsvec(W _i ). Set z=svec(Z),w _i =svec(W _i ),e=svec(I _q−p ). Thus the above problem can be equivalently transformed into

$$\begin{aligned} \min &\quad z^T \Biggl[\sum ^n_{i=1} w_i w^{T}_i \Biggr] z \\ \textrm{s.t.} &\quad e^T z =1. \end{aligned}$$

The problem is the quadratic programming about z, which can be solved through KKT condition. Its Lagrangian function is

$$ L(z,\mu)= z^T W z + \mu\bigl(e^T z - 1\bigr), $$

where $W:= \sum^{n}_{i=1} w_{i} w^{T}_{i}$, the K-T pair (z ^∗,μ ^∗) for the convex quadratic programming should satisfy the following condition

$$ W z^{\ast} +\mu^{\ast} e = 0, \quad e^T z^{\ast} =1, $$

i.e.,

$$ \left (\begin{array}{c@{\quad}c} W & e \\ e^T & 0 \end{array} \right ) \begin{pmatrix} z^{\ast} \\ \mu^{\ast} \\ \end{pmatrix} = \begin{pmatrix} 0 \\ 1 \\ \end{pmatrix} . $$

In fact, since z is dependent on the variable $\hat{x}$, and, according to the result of the previous proof, we get the solution of Z about $\hat{x}$ is unique, so the solution of z about $\hat{x}$ also is single. So this coefficient matrix is invertible in above expression, making using of the method for solving equality-constrained quadratic programming we can solve our problem model.