Alternating direction method of multipliers for a class of nonconvex bilinear optimization: convergence analysis and applications

Abstract

In this paper, we study a class of nonconvex nonsmooth optimization problems with bilinear constraints, which have wide applications in machine learning and signal processing. We propose an algorithm based on the alternating direction method of multipliers, and rigorously analyze its convergence properties (to the set of stationary solutions). To test the performance of the proposed method, we specialize it to the nonnegative matrix factorization problem and certain sparse principal component analysis problem. Extensive experiments on real and synthetic data sets have demonstrated the effectiveness and broad applicability of the proposed methods.

Appendix

1.1 Proof of Lemma 1

Since $\left( X, Z\right) ^{r+1}$ is optimal solution of (22b), it should satisfy the optimality condition which is given by:

$$\begin{aligned} \nabla f(Z^{r+1}) + {\varLambda }^r + \rho (Z^{r+1} - X^{r+1}Y^{r+1}) =0. \end{aligned}$$

(38)

If we combine Eq. (38) with dual update variable (22c), we have:

$$\begin{aligned} {\varLambda }^{r+1} = -\nabla f(Z^{r+1}). \end{aligned}$$

(39)

Applying the Eq. (23), eventually we have

$$\begin{aligned} \Vert {\varLambda }^{r+1}-{\varLambda }^r\Vert _F^2&= \Vert \nabla f(Z^{r+1}) - \nabla f(Z^{r})\Vert _F^2\le L^2\Vert Z^{r+1}-Z^r\Vert _F^2. \end{aligned}$$

(40)

The lemma is proved. Q.E.D.

1.2 Proof of Lemma 2

Part 1 First let us prove that $r_1(X^{r+1})\le u_1(X^{r+1},X^r)$ which are given in (18) and (19), and similarly we can show that $r_2(Y^{r+1})\le u_2(Y^{r+1},Y^r)$. When $r_1$ is convex we have $u_1(X,X^r)=r_1(X)$, so if we set $X=X^{r+1}$ we get $u_1(X^{r+1},X^r)=r_1(X^{r+1})\ge r_1(X^{r+1})$. In the second case when $r_1$ is concave, we have

$$\begin{aligned} u_1(X,X^r)=r_1(X^r) + h'_1(l_1(X^r))\left[ l_1(X) - l_1(X^r)\right] . \end{aligned}$$

For simplicity, let us do the change of variable $Z:=l_1(X)$, therefore we have $r_1(X)=h_1(l_1(X))=h_1(Z).$ Based on the critical property of a concave function (i.e. linear approximation is global upper-estimation for the function) we have for every $Z,\hat{Z}\in \text{ dom }\,(h_1)$

$$\begin{aligned} h_1(Z)\le h_1(\hat{Z}) + h'_1(\hat{Z})(Z-\hat{Z}). \end{aligned}$$

If we plug back in $Z:=l_1(X)$, and $\hat{Z}:=l_1(\hat{X})$ in the above equation, we have

$$\begin{aligned} h_1(l_1(X))\le h_1(l_1(\hat{X}))+ h'_1(l_1(\hat{X}))\big [l_1(X)-l_1(\hat{X})\big ]. \end{aligned}$$

Now if we set $X=X^{r+1}$, and $\hat{X}=X^r$, we reach

$$\begin{aligned} h_1(l_1(X^{r+1}))\le h_1(l_1(X^r))+ h'_1(l_1(X^r))\big [l_1(X^{r+1})-l_1(X^r)\big ] \end{aligned}$$

which is equivalent to $r_1(X^{r+1})\le u_1(X^{r+1},X^r).$

Next, for simplicity let us define $W^r:=(Y^{r}, (X, Z)^{r}; {\varLambda }^{r})$. Then, the successive difference of augmented Lagrangian can be written as the following by adding and subtracting the term $L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]$

$$\begin{aligned} L_\rho [W^{r+1}]-L_\rho [W^{r}]=&L_\rho [W^{r+1}]-L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}] \nonumber \\&+L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]-L_\rho [W^{r}]. \end{aligned}$$

(41)

First we bound $L_\rho [W^{r+1}]-L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]$. Using Lemma 1 we have

$$\begin{aligned}&L_\rho [W^{r+1}]-L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]\nonumber \\&\quad =\langle {\varLambda }^{r+1} - {\varLambda }^{r}, Z^{r+1}-X^{r+1}Y^{r+1}\rangle = \frac{1}{\rho }\Vert {\varLambda }^{r+1} - {\varLambda }^{r}\Vert _F^2 \nonumber \\&\quad {\mathop {\le }\limits ^{(40)}}\frac{L^2}{\rho }\Vert Z^{r+1}-Z^{r}\Vert _F^2. \end{aligned}$$

(42)

Next let us bound $L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]-L_\rho [W^{r}]$.

$$\begin{aligned}&L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]-L_\rho [W^{r}]\nonumber \\&\quad =L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]-L_\rho [Y^{r+1}, (X, Z)^{r}; {\varLambda }^{r}]\nonumber \\&\quad +L_\rho [Y^{r+1}, (X, Z)^{r}; {\varLambda }^{r}]-L_\rho [W^{r}] \end{aligned}$$

(43)

Suppose that $\xi _X\in \partial _X H[(X,Z)^{r+1}; X^r, Y^{r+1}, {\varLambda }^r]$ is a subgradient of function $H[(X,Z); X^r, Y^{r+1}, {\varLambda }^r]$ at the point $X^{r+1}$. Then we have

$$\begin{aligned}&L_\rho [Y^{r+1}, (X, Z)^{r+1}; {\varLambda }^{r}]-L_\rho [Y^{r+1}, (X, Z)^{r}; {\varLambda }^{r}] \nonumber \\&\quad =H[(X,Z)^{r+1}; X^r, Y^{r+1}, {\varLambda }^r]-H[(X,Z)^{r}; X^r, Y^{r+1}, {\varLambda }^r]-\frac{\beta }{2}\Vert X^{r+1}-X^r\Vert _F^2\nonumber \\&\qquad +\left[ r_1(X^{r+1})-u_1(X^{r+1},X^r)\right] \nonumber \\&\quad {\mathop {\le }\limits ^\mathrm{(i)}}H[(X,Z)^{r+1}; X^r, Y^{r+1}, {\varLambda }^r]-H[(X,Z)^{r}; X^r, Y^{r+1}, {\varLambda }^r]-\frac{\beta }{2}\Vert X^{r+1}-X^r\Vert _F^2\nonumber \\&\quad {\mathop {\le }\limits ^\mathrm{(ii)}}\left\langle \xi _X, X^{r+1}-X^r\right\rangle +\langle \nabla _Z H[(X,Z)^{r+1}; X^r, Y^{r+1}, {\varLambda }^r], Z^{r+1}-Z^r \rangle \nonumber \\&\qquad -\frac{\gamma _x}{2}\Vert X^{r+1}-X^r\Vert _F^2-\frac{\gamma _z}{2}\Vert Z^{r+1}-Z^r\Vert _F^2-\frac{\beta }{2}\Vert X^{r+1}-X^r\Vert _F^2\nonumber \\&\quad {\mathop {=}\limits ^\mathrm{(iii)}}-\bigg (\frac{\gamma _x}{2}+\frac{\beta }{2}\bigg )\Vert X^{r+1}-X^r\Vert _F^2-\frac{\gamma _z}{2}\Vert Z^{r+1}-Z^r\Vert _F^2, \end{aligned}$$

(44)

where $\mathrm{(i)}$ is true because from Eqs. (18) and (19) we conclude that $r_1(X^{r+1})\le u_1(X^{r+1},X^{r})$, $\mathrm (ii)$ comes from the strong convexity of $H[(X,Z); X^r, Y^{r+1}, {\varLambda }^r]$ with respect to X and Z with modulus $\gamma _x$ and $\gamma _z$ respectively, and we have $\mathrm{(iii)}$ due to the optimality condition for the problem (22b), which given by

$$\begin{aligned}&\nabla _Z H[(X,Z)^{r+1}; X^r, Y^{r+1}, {\varLambda }^r]=0, \,\text {and}\quad 0\in \partial _X H[(X,Z)^{r+1}; X^r, Y^{r+1}, {\varLambda }^r], \end{aligned}$$

thus, the first term in the second inequality disappears because we implicitly set $\xi _X=0$.

Next suppose $\xi _Y\in \partial _Y G[Y^{r+1}; (X,Z)^r, Y^r,{\varLambda }^r]$. Similarly we bound $L_\rho [Y^{r+1}, (X, Z)^{r}; {\varLambda }^{r}]-L_\rho [W^{r}]$ as follows

$$\begin{aligned} L_\rho [Y^{r+1}, (X, Z)^{r}; {\varLambda }^{r}]-L_\rho [W^{r}]&= G[Y^{r+1}; (X,Z)^r, Y^r,{\varLambda }^r]-G[Y^r; (X,Z)^r, Y^r,{\varLambda }^r]\nonumber \\&\quad -\frac{\beta }{2}\Vert Y^{r+1}-Y^r\Vert _F^2 +\left[ r_2(Y^{r+1})-u_2(Y^{r+1},Y^r)\right] \nonumber \\&{\mathop {\le }\limits ^{\mathrm{(i)}}}G[Y^{r+1}; (X,Z)^r, Y^r,{\varLambda }^r]-G[Y^r; (X,Z)^r, Y^r,{\varLambda }^r]\nonumber \\&\quad -\frac{\beta }{2}\Vert Y^{r+1}-Y^r\Vert _F^2\nonumber \\&{\mathop {\le }\limits ^{\mathrm{(ii)}}}\left\langle \xi _Y, Y^{r+1}-Y^r\right\rangle -\frac{\gamma _y}{2}\Vert Y^{r+1}-Y^r\Vert _F^2\nonumber \\&\quad -\frac{\beta }{2}\Vert Y^{r+1}-Y^r\Vert _F^2\nonumber \\&{\mathop {=}\limits ^{\mathrm{(iii)}}}-\left( \frac{\gamma _y}{2}+\frac{\beta }{2}\right) \Vert Y^{r+1}-Y^r\Vert _F^2, \end{aligned}$$

(45)

where ${\mathrm{(i)}}$ is true because from Eqs. (18) and (19) we conclude that $r_2(Y^{r+1})\le u_2(Y^{r+1},Y^{r})$, ${\mathrm{(ii)}}$ comes from the strong convexity of $G[Y;(X,Z)^r,Y^r, {\varLambda }^r]$ with respect to Y with modulus $\gamma _y$, and we have $\mathrm{(iii)}$ due to the optimality condition for the problem (22a), which is $0\in \partial _YG[Y^{r+1}; (X,Z)^r, Y^r,{\varLambda }^r]$, and we set $\xi _Y=0$. Combining the Eqs. (42), (44) and (45), we have

$$\begin{aligned}&L_\rho [W^{r+1}]-L_\rho [W^{r}]\nonumber \\&\quad \le -\left( \frac{\gamma _{z}}{2}-\frac{L^2}{\rho }\right) \Vert Z^{r+1}-Z^r\Vert ^2_F-\left( \frac{\gamma _y}{2}+\frac{\beta }{2}\right) \Vert Y^{r+1}-Y^r\Vert _F^2\nonumber \\&\quad -\left( \frac{\gamma _x}{2}+\frac{\beta }{2}\right) \Vert X^{r+1}-X^r\Vert ^2_F. \end{aligned}$$

(46)

To complete the proof we only need to set $C_z=\frac{\gamma _z}{2}-\frac{L^2}{\rho }$, $C_y=\frac{\gamma _y+\beta }{2}$, and $C_x=\frac{\gamma _x+\beta }{2}$. Furthermore, since the subproblems (22a) and (22b) are strongly convex with modulus $\gamma _x\ge 0$, $\gamma _y\ge 0$, we have $C_y\ge 0$, and $C_x\ge 0$. Consequently, when $\rho \ge \frac{2L^2}{\gamma _{z}}$ we also have $C_z\ge 0$. Thus, the augmented Lagrangian function is always decreasing.

Part 2 Now we show that the augmented Lagrangian function is lower bounded

$$\begin{aligned} L_\rho [W^{r+1}]&= f(Z^{r+1}) +r_1(X^{r+1})+r_2(Y^{r+1})+ \langle {\varLambda }^{r+1}, Z^{r+1}-X^{r+1}Y^{r+1} \rangle \nonumber \\&\quad +\frac{\rho }{2}\Vert Z^{r+1}-X^{r+1}Y^{r+1}\Vert ^2_F\nonumber \\&{\mathop {=}\limits ^\mathrm{(i)}} f(Z^{r+1})+r_1(X^{r+1})+r_2(Y^{r+1})+\langle \nabla f(Z^{r+1}), X^{r+1}Y^{r+1}-Z^{r+1} \rangle \nonumber \\&\quad +\frac{\rho }{2}\Vert Z^{r+1}-X^{r+1}Y^{r+1}\Vert ^2_F \nonumber \\&{\mathop {\ge }\limits ^\mathrm{(ii)}}f(Z^{r+1})+r_1(X^{r+1})+r_2(Y^{r+1})+\langle \nabla f(Z^{r+1}), X^{r+1}Y^{r+1}-Z^{r+1} \rangle \nonumber \\&\quad +\frac{L}{2}\Vert Z^{r+1}-X^{r+1}Y^{r+1}\Vert ^2_F \nonumber \\&{\mathop {\ge }\limits ^\mathrm{(iii)}}f(X^{r+1}Y^{r+1}) + r_1(X^{r+1})+r_2(Y^{r+1}) \nonumber \\&= g(X^{r+1}, Y^{r+1}, X^{r+1}Y^{r+1}), \end{aligned}$$

(47)

where in $\mathrm (i)$ we use Eq. (39), $\mathrm{(ii)}$ is true because we have picked $\rho \ge L$, and $\mathrm{(iii)}$ comes from the fact that for Lipchitz continuous function f we have

$$\begin{aligned} f(x)\le f(y)+\langle \nabla f(y), x-y \rangle +\frac{L}{2}\Vert x-y\Vert ^2\quad \forall x,y \in \text {Dom(f)}. \end{aligned}$$

Due to the lower boundedness of g(X, Y, Z) (Assumption A) we have $g(X^{r+1}, Y^{r+1}, X^{r+1}Y^{r+1})\ge \underline{g}$. Together with Eq. (47), we can set $\underline{L}=\underline{g}$.

The proof is complete. Q.E.D.

1.3 Proof of Theorem 1

Part 1 Clearly, Eq. (29) together with the fact that augmented Lagrangian is lower bounded imply

$$\begin{aligned} Z^{r+1}-Z^r\rightarrow 0;~ Y^{r+1}-Y^r\rightarrow 0;~ X^{r+1}-X^r\rightarrow 0. \end{aligned}$$

(48)

Further, applying Lemma (1) we have

$$\begin{aligned} {\varLambda }^{r+1}-{\varLambda }^r\rightarrow 0. \end{aligned}$$

Utilizing the update equation for dual variable (22c), eventually we obtain

$$\begin{aligned} X^{r+1}Y^{r+1}-Z^{r+1}\rightarrow 0 . \end{aligned}$$

(49)

The part (1) is proved. From this part condition (26d) can be derived.

Part 2 Since $Y^{r+1}$ minimizes problem (22a), we have

$$\begin{aligned} 0\in h'_2(l_2(Y^r))\partial l_2(Y^{r+1})-\rho (X^r)^\top \left( Z^r-X^{r}Y^{r+1}+\frac{1}{\rho }{\varLambda }^r\right) +\beta \left( Y^{r+1}-Y^r\right) . \end{aligned}$$

Thus, there exists $\xi \in \partial l_2(Y^{r+1})$ such that for every Y

$$\begin{aligned} \langle Y-Y^{r+1} , {\varPhi }_2^r\xi - \rho (X^r)^\top \left( Z^r-X^{r}Y^{r+1}+\frac{1}{\rho }{\varLambda }^r\right) +\beta (Y^{r+1}-Y^r) \rangle \ge 0, \end{aligned}$$

(50)

where we have set ${\varPhi }_2^r = h'_2(l_2(Y^r))$ for notational simplicity. Because $l_2(Y)$ is a convex function we have the following inequality for every $\xi \in \partial l_2(Y^{r+1})$

$$\begin{aligned} l_2(Y) - l_2(Y^{r+1})\ge \langle \xi , Y- Y^{r+1}\rangle ;\quad \forall ~Y. \end{aligned}$$

(51)

Further since we assumed that $h_2$ is non-decreasing, we have $h'_2(l_2(Y^r))\ge 0$. Combining this fact with (51), yields the following

$$\begin{aligned} {\varPhi }_2^rl_2(Y) - {\varPhi }_2^rl_2(Y^{r+1})\ge \langle {\varPhi }_2^r\xi , Y- Y^{r+1}\rangle ;\quad \forall ~Y. \end{aligned}$$

(52)

If we plug Eqs. (52) into (50) we obtain

$$\begin{aligned}&{\varPhi }_2^rl_2(Y) - {\varPhi }_2^rl_2(Y^{r+1}) \nonumber \\&+ \bigg \langle Y^{r+1}-Y , \rho (X^r)^\top (Z^r-X^{r}Y^{r+1}+\frac{1}{\rho }{\varLambda }^r)+\beta (Y^{r+1}-Y^r)\bigg \rangle \ge 0. \end{aligned}$$

(53)

Next taking the limit over (53) and utilizing the facts that $\Vert Y^{r+1}-Y^r\Vert \rightarrow 0$, and $\Vert X^{r+1}Y^{r+1}-Z^{r+1}\Vert _F\rightarrow 0$, we obtain the following

$$\begin{aligned} {\varPhi }_2^*l_2(Y) - {\varPhi }_2^*l_2(Y^{*})+ \langle Y^*-Y , (X^*)^\top {\varLambda }^* \rangle \ge 0;\quad \forall ~Y. \end{aligned}$$

(54)

where ${\varPhi }_2^* = h'_2(l_2(Y^*))$. From Eq. (54) we can conclude that

$$\begin{aligned}&{\varPhi }_2^*l_2(Y) + \langle Y^*-Y , (X^*)^\top {\varLambda }^* \rangle \ge {\varPhi }_2^*l_2(Y^{*})+\langle Y^*-Y^{*} , (X^*)^\top {\varLambda }^* \rangle ; \quad \forall ~Y, \end{aligned}$$

(55)

which further implies

$$\begin{aligned} Y^*\in \mathop {\mathrm{argmin}}_{Y}\left( {\varPhi }_2^*l_2(Y)+\langle Y^*-Y, (X^*)^\top {\varLambda }^* \rangle \right) . \end{aligned}$$

This is equivalent to

$$\begin{aligned} (X^*)^\top {\varLambda }^*\in h'_2\left( l_2(Y^*)\right) \partial l_2(Y^*). \end{aligned}$$

Applying the chain rule for Clark sub-differential given in Eq. (27), we have

$$\begin{aligned} (X^*)^\top {\varLambda }^*\in \partial ^c (h_2\circ l_2)(Y^*) = \partial ^c r_2(Y^*). \end{aligned}$$

From this equation we can simply conclude that

$$\begin{aligned} \text {dist}(\partial ^c r_2(Y^*),(X^*)^\top {\varLambda }^*)=0 \end{aligned}$$

This proves the Eq. (26c).

Now let us consider (X, Z) step (22b). Similarly let us define ${\varPhi }_1 = h'_1(l_1(X^{r}))$. Since $(X,Z)^{r+1}$ minimizes (22b), we have

$$\begin{aligned}&\nabla f(Z^{r+1})+{\varLambda }^{r+1}+\rho \left( Z^{r+1}-X^{r+1}Y^{r+1}\right) = 0; \end{aligned}$$

(56)

$$\begin{aligned}&\quad 0\in {\varPhi }_1\partial l_1(X^{r+1})-\rho \left( Z^{r+1}-X^{r+1}Y^{r+1}+\frac{1}{\rho }{\varLambda }^r\right) (Y^{(r+1)\top })+\beta (X^{r+1}-X^r). \end{aligned}$$

(57)

Let us take limit over the Eqs. (56) and (57). Then invoking (48, and (49) and following the same process for proving (26b) it follows that

$$\begin{aligned}&\nabla f(Z^*)+{\varLambda }^* = 0; \end{aligned}$$

(58)

$$\begin{aligned}&\quad \text {dist}\left( \partial ^c r_1(X^*),{\varLambda }^*(Y^*)^\top \right) =0 \end{aligned}$$

(59)

which verifies (26a) and (26c).

The theorem is proved. Q.E.D.