Stochastic Primal Dual Fixed Point Method for Composite Optimization

Abstract

In this paper we propose a stochastic primal dual fixed point method for solving the sum of two proper lower semi-continuous convex function and one of which is composite. The method is based on the primal dual fixed point method proposed in Chen et al. (Inverse Probl 29:025011, 2013) that does not require subproblem solving. Under some mild condition, the convergence is established based on two sets of assumptions: bounded and unbounded gradients and the convergence rate of the expected error of iterate is of the order ${\mathcal {O}}(k^{-\alpha })$ where k is iteration number and $\alpha \in (0,1]$. Finally, numerical examples on graphic Lasso and logistic regressions are given to demonstrate the effectiveness of the proposed algorithm.

Appendix

Here, we give the details of the aforementioned lemmas. First, we give a lemma that will be used in the proof of Lemma 3.2.

Lemma 7.1

Letting $r > 0$ and $h(x) = r f_0(x/r),x \in {\mathbb {R}}^d$, then, for all$~ y \in {\mathbb {R}}^d$,

$$\begin{aligned} {\mathrm {Prox}}_h(y) = r {\mathrm {Prox}}_{r^{-1}f_0}(y/r). \end{aligned}$$

Proof

The assertion can be proved by using the definition of ${\mathrm {Prox}}_{f_0}(\cdot )$ and change of variables.

$\square$

1.1 Proof of Lemma 3.2

Proof

By the first optimality condition of problem (1.1), we have

$$\begin{aligned} \begin{aligned} x^{*}&= \underset{x \in {\mathbb {R}}^d}{\arg \min } (f_1 \circ B)(x) + f_2(x) \\&\Leftrightarrow 0 \in -\gamma _k \nabla f_2(x^*) - \gamma _k \partial (f_1 \circ B )(x^*) \\&\Leftrightarrow x^* \in x^* -\gamma _k \nabla f_2(x^*) - \gamma _k B^T \partial f_1(Bx^*) \\&\Leftrightarrow x^* \in x^* -\gamma _k \nabla f_2(x^*) - \lambda B^T \circ \frac{\gamma _k}{\lambda } \partial f_1(Bx^*). \end{aligned} \end{aligned}$$

(7.1)

Letting

$$\begin{aligned} v^* \in \partial f_1(Bx^*), \end{aligned}$$

(7.2)

then Eq. (7.1) can be rewritten as

$$\begin{aligned} x^* = x^* - \gamma _k \nabla f_2(x^*) - \gamma _k B^T v^* . \end{aligned}$$

(7.3)

From Eq. (7.2), we also have

$$\begin{aligned} \frac{\gamma _k}{\lambda }v^* \in \partial \frac{\gamma _k}{\lambda }f_1(Bx^*), \end{aligned}$$

(7.4)

which means that

$$\begin{aligned} \begin{aligned} Bx^*&= {\mathrm {Prox}}_{\frac{\gamma _k}{\lambda }f_1}\left( Bx^* + \frac{\gamma _k}{\lambda }v^*\right) \\&\Leftrightarrow \left( Bx^* + \frac{\gamma _k}{\lambda }v^*\right) - \frac{\gamma _k}{\lambda }v^* = {\mathrm {Prox}}_{\frac{\gamma _k}{\lambda }f_1}\left( Bx^* + \frac{\gamma _k}{\lambda }v^*\right) \\&\Leftrightarrow \frac{\gamma _k}{\lambda }v^* = \left( I - {\mathrm {Prox}}_{\frac{\gamma _k}{\lambda }f_1}\right) \left( Bx^* + \frac{\gamma _k}{\lambda }v^*\right) \\&\Leftrightarrow v^* = \frac{\lambda }{\gamma _k}\left( I - {\mathrm {Prox}}_{\frac{\gamma _k}{\lambda }f_1}\right) \left( Bx^* + \frac{\gamma _k}{\lambda }v^*\right) \\&\Leftrightarrow v^* = \frac{\lambda }{\gamma _k}Bx^* + v^* - \frac{\lambda }{\gamma _k}{\mathrm {Prox}}_{\frac{\gamma _k}{\lambda }f_1}\left( Bx^* + \frac{\gamma _k}{\lambda }v^*\right) \\&\Leftrightarrow v^* = \frac{\lambda }{\gamma _k}Bx^* + v^* - \frac{\lambda }{\gamma _k}{\mathrm {Prox}}_{\left( \frac{\lambda }{\gamma _k}\right) ^{-1}f_1}\Big (\frac{\frac{\lambda }{\gamma _k}Bx^* + v^*}{\frac{\lambda }{\gamma _k}}\Big ) \\&\Leftrightarrow v^* = \frac{\lambda }{\gamma _k}Bx^* + v^* - {\mathrm {Prox}}_{h^k}\Big (\frac{\lambda }{\gamma _k}Bx^* + v^*\Big ) \\&\Leftrightarrow v^* = (I- {\mathrm {Prox}}_{h^k})\Big (\frac{\lambda }{\gamma _k}Bx^* + v^*\Big ) \end{aligned} , \end{aligned}$$

(7.5)

where, in the second-to-last equality, we let $h^k(x) = \frac{\lambda }{\gamma _k}f_1(\frac{x}{\frac{\lambda }{\gamma _k}}) = \frac{\lambda }{\gamma _k}f_1(\frac{\gamma _k}{\lambda }x)$ and using Lemma 7.1.

Inserting Eq. (7.3) into the last equality of Eq. (7.5), we have

$$\begin{aligned} \begin{aligned} v^*&= (I - {\mathrm {Prox}}_{h^k})\Big (\frac{\lambda }{\gamma _k}B\left( x^* - \gamma _k \nabla f_2(x^*)\right) + \left( I - \lambda BB^T\right) v^* \Big ) \\&\Leftrightarrow v^* = \frac{\lambda }{\gamma _k}\left( I - {\mathrm {Prox}}_{\frac{\gamma _k}{\lambda }f_1}\right) \Big (B\left( x^* - \gamma _k \nabla f_2(x^*)\right) + \left( I - \lambda BB^T\right) \frac{\gamma _k}{\lambda }v^* \Big ). \end{aligned} \end{aligned}$$

(7.6)

Combining (7.3), (7.5) and (7.6), we obtain

$$\begin{aligned} \left\{ \begin{aligned} v^*&= (I - {\mathrm {Prox}}_{h^k})\Big (\frac{\lambda }{\gamma _k}B\left( x^* - \gamma _k \nabla f_2(x^*)\right) + \left( I - \lambda BB^T\right) v^* \Big ) \\&= T_0^{(k)}(x^*,v^*) \\ x^*&= x^* - \gamma _k \nabla f_2(x^*) - \gamma _k B^T T_0^{(k)}(x^*,v^*). \end{aligned} \right. \end{aligned}$$

The converse can be similarly verified. This completes the proof. $\square$

1.2 Proof of Lemma 4.1

Proof

Letting $(x^*,v^*)$ be that in Lemma 3.2 and $(x_k,v_k)$ be the iterate in Algorithm 1, $T_1^{(k)}(\cdot ) = (I - {\mathrm {Prox}}_{h^k})(\cdot )$, where $h^k(x) = \frac{\lambda }{\gamma _k}f_1(\frac{x}{\frac{\lambda }{\gamma _k}}) = \frac{\lambda }{\gamma _k}f_1(\frac{\gamma _k}{\lambda }x)$. We denote

$$\begin{aligned} \begin{aligned} \varphi _{1,i}^{(k)}(x,y)&= \frac{\lambda }{\gamma _k}B(x - \gamma _k\nabla f_2^{[i]}(x))+ (I - \lambda B B^T)y \\&= \frac{\lambda }{\gamma _k}Bg_{k,i}^{(1)}(x) + My \\ \varphi _2^{(k)}(x,y)&= \frac{\lambda }{\gamma _k}B(x - \gamma _k\nabla f_2(x))+ (I - \lambda B B^T)y \\&= \frac{\lambda }{\gamma _k}Bg_k^{(2)}(x) + My \\ \end{aligned} \end{aligned}$$

Here, $g_{k,i}^{(1)}(x) = x - \gamma _k\nabla f_2^{[i]}(x)$ and $g_{k}^{(2)}(x) = x - \gamma _k\nabla f_2(x)$, $M = I - \lambda B B^T$.

• (i) Estimation of $\Vert v_{k + 1} - v^* \Vert _2^2$:

  $$\begin{aligned} \begin{aligned} \Vert v_{k + 1} - v^* \Vert _2^2&= \left\Vert T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}(x_k,v_k)\right) - v^* \right\Vert _2^2 \\&= \left\Vert T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}(x_k,v_k)\right) - T_1^{(k)}\left( \varphi _2^{(k)}(x^*,v^*)\right) \right\Vert _2^2 \\&\le \big< T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}(x_k,v_k)\right) - T_1^{(k)}\left( \varphi _2^{(k)}(x^*,v^*)\right) ,\varphi _{1,i_k}^{(k)}(x_k,v_k) - \varphi _2^{(k)}(x^*,v^*) \big> \\&= \frac{\lambda }{\gamma _k}\big< T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}(x_k,v_k)\right) - T_1^{(k)}\left( \varphi _2^{(k)}(x^*,v^*)\right) ,B\left( g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*)\right) \big> \\&\quad + \big < T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}(x_k,v_k)\right) - T_1^{(k)}\left( \varphi _2^{(k)}(x^*,v^*)\right) , M(v_k - v^*) \big >. \end{aligned} \end{aligned}$$

  (7.7)

  The second equality follows from Eq. (3.2) and the inequality follows from the firm non-expansiveness of $T_1^{(k)}$ (see Definition 2.3).

  Here, and in what follows, for convenience, we denote $T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) = T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}(x_k,v_k)\right)$ and $T_1^{(k)}\left( \varphi _2^{(k)}\right) = T_1^{(k)}\left( \varphi _2^{(k)}(x^*,v^*)\right)$.

• (ii) Estimation of $\Vert x_{k + 1} - x^* \Vert _2^2:$

  $$\begin{aligned}&\Vert x_{k + 1} - x^* \Vert _2^2 \nonumber \\&= \left\Vert x_k - \gamma _k \nabla f_2^{[i_k]}(x_k) - \gamma _k B^T \circ T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - \big (x^* - \gamma _k \nabla f_2(x^*) - \gamma _k B^T \circ T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ) \right\Vert _2^2 \nonumber \\&= \left\Vert g_{k,i_k}^{(1)}(x_k) - \gamma _k B^T \circ T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - \big (g_k^{(2)}(x^*) - \gamma _k B^T \circ T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ) \right\Vert _2^2 \nonumber \\&= \left\Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) - \gamma _k B^T \circ \big (T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \big )\right\Vert _2^2 \nonumber \\&= \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 - 2\gamma _k \big< B^T \circ \big ( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ), g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \big>\nonumber \\&\quad + \frac{\gamma _k^2}{\lambda ^2}\left\Vert \lambda B^T \circ \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _2^2 \nonumber \\&= \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 - 2\gamma _k \big < B^T \circ \big ( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ), g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \big > \nonumber \\&\quad - \frac{\gamma _k^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _M^2 + \frac{\gamma _k^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _2^2. \end{aligned}$$

  (7.8)

  The first equality follows from Eq. (3.2). In the last equality, we use the definition $M = I - \lambda BB^T$ and $\Vert y \Vert _M = \sqrt{<y,My>}$.

• (iii) From (7.7) and (7.8), we have

  $$\begin{aligned}&\Vert x_{k + 1} - x^* \Vert _2^2 + \frac{\gamma _{k + 1}^2}{\lambda }\Vert v_{k + 1} - v^* \Vert _2^2 \nonumber \\&= \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 - 2\gamma _k \big< B^T \circ \big ( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ), g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \big> \nonumber \\&\quad - \frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _M^2 + \frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _2^2 \nonumber \\&\quad + \frac{\gamma _{k + 1}^2}{\lambda }\left\Vert \left( T_1\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _2^2 \nonumber \\&\le \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 - 2\gamma _k \big< B^T \circ \big ( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ), g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \big> \nonumber \\&\quad - \frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _M^2 + 2\frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _2^2 \nonumber \\&\le \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 - \underline{2\gamma _k \big< B^T \circ \big ( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ), g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \big>} \nonumber \\&\quad - \frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _M^2 \nonumber \\&\quad + \underline{2\frac{\gamma _{k}^2}{\lambda } \frac{\lambda }{\gamma _k}\big< T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) ,B\left( g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*)\right) \big>}\nonumber \\&\quad + 2\frac{\gamma _{k}^2}{\lambda } \big< T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) , M(v_k - v^*) \big>\nonumber \\&= \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 - \underline{2\gamma _k \big< B^T \circ \big ( T_1^{(k)}\left( \varphi _1^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \big ), g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \big>} \nonumber \\&\quad - \frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _M^2\nonumber \\&\quad + \underline{2\gamma _{k}\big< T_1^{(k)}\left( \varphi _1^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) ,B\left( g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*)\right) \big>} \nonumber \\&\quad + 2\frac{\gamma _{k}^2}{\lambda } \big< T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) , M(v_k - v^*) \big>\nonumber \\&= \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 + 2\frac{\gamma _{k}^2}{\lambda }\big < T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) , M(v_k - v^*) \big > \nonumber \\&\quad - \frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) \right\Vert _M^2 \nonumber \\&= \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2 + \frac{\gamma _{k}^2}{\lambda }\Vert v_k - v^* \Vert _M^2 \nonumber \\&\quad - \frac{\gamma _{k}^2}{\lambda }\left\Vert \left( T_1^{(k)}\left( \varphi _{1,i_k}^{(k)}\right) - T_1^{(k)}\left( \varphi _2^{(k)}\right) \right) - (v_k - v^*) \right\Vert _M^2 \nonumber \\&\le \left\Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \left\Vert _2^2 + \frac{\gamma _{k}^2}{\lambda }\left( 1 - \lambda \rho _{min}(B B^T)\right) \right\Vert v_k - v^* \right\Vert _2^2, \end{aligned}$$

  (7.9)

  where the first equality uses Eq. (7.8) and the second equality of (7.7). The first inequality uses the fact that $\gamma _k$ is decreasing with respect to k. The second inequality uses Eq. (7.7). The last inequality uses the fact that $0 < \lambda \le \frac{1}{\rho _{max}(B B^T)}$, which means that $0 \preceq M \preceq (1 - \lambda \rho _{min}(B B^T))$.

Considering the expectations of both sides of inequality (7.9), we obtain

$$\begin{aligned}&{\mathbb {E}}^{(k + 1)}\left( \Vert x_{k + 1} - x^* \Vert _2^2 + \frac{\gamma _{k+1}^2}{\lambda }\Vert v_{k + 1} - v^* \Vert _2^2 \right) \nonumber \\&\le {\mathbb {E}}^{(k + 1)}\left( \Vert g_{k,i_k}^{(1)}(x_k) - g_k^{(2)}(x^*) \Vert _2^2\right) + \frac{\gamma _{k}^2}{\lambda }\left( 1 - \lambda \rho _{min}(B B^T)\right) {\mathbb {E}}^{(k)}\left( \Vert v_k - v^* \Vert _2^2 \right) \nonumber \\&= {\mathbb {E}}^{(k + 1)}\Big (\Vert x_k - x^* - \gamma _k\left( \nabla f_2^{[i_k]}(x_k) - \nabla f_2(x^*)\right) \Vert _2^2 \Big ) \nonumber \\&\quad + \frac{\gamma _{k}^2}{\lambda }\left( 1 - \lambda \rho _{min}(B B^T)\right) {\mathbb {E}}^{(k)}\left( \Vert v_k - v^* \Vert _2^2 \right) \nonumber \\&= {\mathbb {E}}^{(k)}\big (\Vert x_k - x^* \Vert _2^2\big ) + \frac{\gamma _{k}^2}{\lambda }\left( 1 - \lambda \rho _{min}(B B^T)\right) {\mathbb {E}}^{(k)}\left( \Vert v_k - v^* \Vert _2^2 \right) \nonumber \\&\quad - 2\gamma _k {\mathbb {E}}^{(k)}\big < \nabla f_2(x_k) - \nabla f_2(x^*), x_k - x^*\big > + \gamma _k^2 {\mathbb {E}}^{(k + 1)}\left( \Vert \nabla f_2^{[i_k]}(x_k) - \nabla f_2(x^*) \Vert _2^2\right) , \end{aligned}$$

(7.10)

where, in the third term of the last equality, we use the fact that

$$\begin{aligned} {\mathbb {E}}^{(k + 1)} \big (\nabla f_2^{[i_k]}(x_k)\big ) = {\mathbb {E}}^{(k)}\big (\nabla f_2(x_k)\big ). \end{aligned}$$

This completes the proof. $\square$