Accelerated Doubly Stochastic Gradient Descent for Tensor CP Decomposition

Abstract

In this paper, we focus on the acceleration of doubly stochastic gradient descent method for computing the CANDECOMP/PARAFAC (CP) decomposition of tensors. This optimization problem has N blocks, where N is the order of the tensor. Under the doubly stochastic framework, each block subproblem is solved by the vanilla stochastic gradient method. However, the convergence analysis requires that the variance converges to zero, which is hard to check in practice and may not hold in some implementations. In this paper, we propose accelerating the stochastic gradient method by the momentum acceleration and the variance reduction technique, denoted as DS-MVR. Theoretically, the convergence of DS-MVR only requires the variance to be bounded. Under mild conditions, we show DS-MVR converges to a stochastic $\varepsilon$-stationary solution in $\tilde{\mathcal {O}}(N^{3/2}\varepsilon ^{-3})$ iterations with varying stepsizes and in $\mathcal {O}(N^{3/2}\varepsilon ^{-3})$ iterations with constant stepsizes, respectively. Numerical experiments on four real-world datasets show that our proposed algorithm can get better results compared with the baselines.

Appendix A: Several Lemmas for Theoretical Analysis in Algorithm 1

Lemma A.1

Under Assumption 4.3, we have

$$\begin{aligned} \begin{aligned}&\mathbb {E}_{\zeta ^{k}}\Big [\Big \Vert G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^k+(1-\beta ^{k-1})\Big (\nabla _{A_{\xi ^{k}}}f^{k-1}-G_{\xi ^{k}}^{k-1}\Big )\Big \Vert ^{2}\vert \mathcal {B}^{k},\xi ^{k}\Big ]\\ \le&2(\beta ^{k-1})^{2}\mathbb {E}_{\zeta ^{k}}\left[ \left\| G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^k\right\| ^{2}\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+2(1-\beta ^{k-1})^{2} \mathbb {E}_{\zeta ^{k}}\left[ \left\| G_{\xi ^{k}}^{k}- G_{\xi ^{k}}^{k-1} \right\| ^{2}\Big \vert \mathcal {B}^{k},\xi ^{k}\right] , \end{aligned} \end{aligned}$$

(40)

where $\nabla _{A_{\xi ^{k}}}f^k=\nabla _{A_{\xi ^{k}}}f\left( A_{1}^{k},\dots ,A_{N}^{k}\right)$ and $\nabla _{A_{\xi ^{k}}}f^{k-1} = \nabla _{A_{\xi ^{k}}}f\left( A_{1}^{k-1},\dots ,A_{N}^{k-1}\right)$.

Proof

We have the following inequality holds

$$\begin{aligned} \begin{aligned}&\left\| G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^k+(1-\beta ^{k-1})\left( \nabla _{A_{\xi ^{k}}}f^{k-1}-G_{\xi ^{k}}^{k-1}\right) \right\| ^{2}\\ =&\quad \Big \Vert (1-\beta ^{k-1})\left( \nabla _{A_{\xi ^{k}}}f^{k-1}-G_{\xi ^{k}}^{k-1}+G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^k\right) +\beta ^{k-1}\left( G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^k\right) \Big \Vert ^{2}\\ \le&\quad 2(1-\beta ^{k-1})^{2}\left\| \nabla _{A_{\xi ^{k}}}f^{k-1}-G_{\xi ^{k}}^{k-1}+G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^k\right\| ^{2}\\&\quad +2(\beta ^{k-1})^{2}\left\| G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^k\right\| ^{2}, \end{aligned}\nonumber \\ \end{aligned}$$

(41)

where the last inequality follows from Young’s inequality. From Assumption 4.3, we have that

$$\begin{aligned}&\mathbb {E}_{\zeta ^{k}}\left[ \left\| \nabla _{A_{\xi ^{k}}}f^{k-1}-G_{\xi ^{k}}^{k-1}+G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^{k}\right\| ^{2}\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&=\mathbb {E}_{\zeta ^{k}}\left[ \left\| \nabla _{A_{\xi ^{k}}}f^{k-1}-G_{\xi ^{k}}^{k-1}\right\| ^{2}+\left\| G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^{k}\right\| ^{2}\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+2\mathbb {E}_{\zeta ^{k}}\left[ \left<\nabla _{A_{\xi ^{k}}}f^{k-1}-G_{\xi ^{k}}^{k-1},G_{\xi ^{k}}^{k}-\nabla _{A_{\xi ^{k}}}f^{k}\right>\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&=\mathbb {E}_{\zeta ^{k}}\left[ \left\| \nabla _{A_{\xi ^{k}}}f^{k-1}\right\| ^{2}+\left\| G_{\xi ^{k}}^{k-1}\right\| ^{2}\Big \vert \mathcal {B}^{k},\xi ^{k}\right] -2\mathbb {E}_{\zeta ^{k}}\left[ \left<\nabla _{A_{\xi ^{k}}}f^{k-1},G_{\xi ^{k}}^{k-1}\right>\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+\mathbb {E}_{\zeta ^{k}}\left[ \left\| \nabla _{A_{\xi ^{k}}}f^{k}\right\| ^{2}+\left\| G_{\xi ^{k}}^{k}\right\| ^{2}-2\left<\nabla _{A_{\xi ^{k}}}f^{k},G_{\xi ^{k}}^{k}\right>\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+2\mathbb {E}_{\zeta ^{k}}\left[ -\left<\nabla _{A_{\xi ^{k}}}f^{k-1},\nabla _{A_{\xi ^{k}}}f^{k}\right>\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+2\mathbb {E}_{\zeta ^{k}}\left[ -\left<G_{\xi ^{k}}^{k-1},G_{\xi ^{k}}^{k}\right>+\left<G_{\xi ^{k}}^{k-1},\nabla _{A_{\xi ^{k}}}f^{k}\right>\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+2\mathbb {E}_{\zeta ^{k}}\left[ \left<\nabla _{A_{\xi ^{k}}}f^{k-1},G_{\xi ^{k}}^{k}\right>\vert \mathcal {B}^{k},\xi ^{k}\right] \\&=\mathbb {E}_{\zeta ^{k}}\left[ \left\| \nabla _{A_{\xi ^{k}}}f^{k-1}\right\| ^{2}+\left\| G_{\xi ^{k}}^{k-1}\right\| ^{2}-2\Vert \nabla _{A_{\xi ^{k}}}f^{k-1}\Vert ^2 \Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+\mathbb {E}_{\zeta ^{k}}\left[ \left\| \nabla _{A_{\xi ^{k}}}f^{k}\right\| ^{2}+\left\| G_{\xi ^{k}}^{k}\right\| ^{2}-2\Vert \nabla _{A_{\xi ^{k}}}f^{k}\Vert ^2 \Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&+2\mathbb {E}_{\zeta ^{k}}\left[ -\left<G_{\xi ^{k}}^{k-1},G_{\xi ^{k}}^{k}\right>+\left<\nabla _{A_{\xi ^{k}}}f^{k-1},\nabla _{A_{\xi ^{k}}}f^{k}\right>\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&=\mathbb {E}_{\zeta ^{k}}\left[ \left\| G_{\xi ^{k}}^{k}- G_{\xi ^{k}}^{k-1} \right\| ^{2}-\left\| \nabla _{A_{\xi ^{k}}}f^{k}-\nabla _{A_{\xi ^{k}}}f^{k-1}\right\| ^{2}\Big \vert \mathcal {B}^{k},\xi ^{k}\right] \\&\le \mathbb {E}_{\zeta ^{k}}\left[ \left\| G_{\xi ^{k}}^{k}- G_{\xi ^{k}}^{k-1} \right\| ^{2}\Big \vert \mathcal {B}^{k},\xi ^{k}\right] , \end{aligned}$$

where the third equality follows from Assumption 4.3. $\square$

Lemma A.2

Suppose the assumption in Theorem 4.1 holds. Then, we have

$$\begin{aligned} \begin{aligned}&76^{2}\frac{2L^{4}}{\eta }\sum _{k=0}^{K-1}(k+4)^{\frac{1}{3}}(\eta ^{k})^{4}+\frac{2}{\eta }\sum _{k=0}^{K-1}(k+4)^{\frac{1}{3}}\left( 1-\left( \frac{k+3}{k+4}\right) ^{\frac{1}{3}}\right) ^{2}\\ \le&12938\eta ^{3}L^{4}\log (K+2)+\frac{1}{3\eta }. \end{aligned} \end{aligned}$$

(42)

Proof

It shows that

$$\begin{aligned} \begin{aligned} \sum _{k=0}^{K-1}(k+4)^{\frac{1}{3}}(\eta ^{k})^{4}=&\eta ^{4}\sum _{k=0}^{K-1}(k+4)^{\frac{1}{3}}(k+3)^{-\frac{4}{3}}\\ =&\eta ^{4}\sum _{k=0}^{K-1}\left( \frac{k+4}{k+3}\right) ^{\frac{1}{3}}(k+3)^{-1}\\ \le&\eta ^{4}\left( \frac{4}{3}\right) ^{\frac{1}{3}}\sum _{k=0}^{K-1}(k+3)^{-1}\\ \le&\eta ^{4}\left( \frac{4}{3}\right) ^{\frac{1}{3}}\log (K+2). \end{aligned} \end{aligned}$$

(43)

Further, we have

$$\begin{aligned} \begin{aligned} 1-\left( \frac{k+3}{k+4}\right) ^{\frac{1}{3}}=&(k+4)^{-\frac{1}{3}}((k+4)^{\frac{1}{3}}-(k+3)^{\frac{1}{3}})\\ =&\frac{(k+4)^{-\frac{1}{3}}}{(k+4)^{\frac{2}{3}}+(k+4)^{\frac{1}{3}}(k+3)^{\frac{1}{3}}+(k+3)^{\frac{2}{3}}}, \end{aligned} \end{aligned}$$

(44)

where the last inequality follows from $a^{3}-b^{3}=(a-b)(a^{2}+ab+b^{2})$ for any $a,b\in \mathbb {R}$. From the above equation, we have

$$\begin{aligned} \begin{aligned}&\sum _{k=0}^{K-1}(k+4)^{\frac{1}{3}}\left( 1-\left( \frac{k+3}{k+4}\right) ^{\frac{1}{3}}\right) ^{2}\\ =&\sum _{k=0}^{K-1}\frac{1}{(k+4)^{\frac{1}{3}}\left( (k+4)^{\frac{2}{3}}+(k+4)^{\frac{1}{3}}(k+3)^{\frac{1}{3}}+(k+3)^{\frac{2}{3}}\right) ^{2}}\\ \le&\frac{1}{9}\sum _{k=0}^{K-1}(k+3)^{-\frac{5}{3}}\\ \le&\frac{1}{6}. \end{aligned} \end{aligned}$$

(45)

where the last inequality follows from $\sum _{k=1}^{+\infty }\frac{1}{k^{\frac{3}{2}}}\approx 2.612$.

Combining (43) and (45), we obtain (42). This completes the proof.

$\square$