Accelerating variance-reduced stochastic gradient methods

Variance reduction is a crucial tool for improving the slow convergence of stochastic gradient descent. Only a few variance-reduced methods, however, have yet been shown to directly benefit from Nesterov’s acceleration techniques to match the convergence rates of accelerated gradient methods. Such approaches rely on “negative momentum”, a technique for further variance reduction that is generally specific to the SVRG gradient estimator. In this work, we show for the first time that negative momentum is unnecessary for acceleration and develop a universal acceleration framework that allows all popular variance-reduced methods to achieve accelerated convergence rates. The constants appearing in these rates, including their dependence on the number of functions n, scale with the mean-squared-error and bias of the gradient estimator. In a series of numerical experiments, we demonstrate that versions of SAGA, SVRG, SARAH, and SARGE using our framework significantly outperform non-accelerated versions and compare favourably with algorithms using negative momentum.


Introduction
We are interested in solving the following composite convex minimisation problem: Throughout, we assume f i : R m → R are convex and have L-Lipschitz continuous gradients for all i.We also assume g : R m → R ∪ {∞} is proper, lower semicontinuous, and µ-strongly convex with µ ≥ 0, but we do not require g to be differentiable.Problems of this form are ubiquitous in many fields, including machine learning, compressed sensing, and image processing (see, e.g., [9,10,21,31]).Fundamental examples include LASSO [31] and matrix completion [10], where f is a least-squares loss and g is the 1 or nuclear norm, respectively, and sparse logistic regression, where f is the logistic loss and g is the 1 norm.One well-studied algorithm that solves (1) is the forward-backward splitting algorithm [11,25].This method has a worst-case convergence rate of O (1/T ) when F is not strongly convex, and when F is µstrongly convex, it converges linearly with a rate of O (1 + κ −1 ) −T , where κ def = L/µ is the condition number of F .The inertial forward-backward splitting algorithm [7] converges at an even faster rate of O 1/T 2 without strong convexity and a linear rate of O (1 + κ −1/2 ) −T when F is strongly convex.The inertial forward-backward method is able to achieve these optimal convergence rates because it incorporates momentum, using information from previous iterates to adjust the current iterate.
Although the inertial forward-backward algorithm converges quickly, it requires access to the full gradient ∇f at each iteration, which can be costly, for instance, when n is large.In many applications, common problem sizes are so large that computing ∇f is prohibitively expensive.Stochastic gradient methods exploit the separable structure of f , using the gradient of a few of the components ∇f i to estimate the full gradient at the current iterate.In most cases, the complexity of computing ∇f i for one i is 1/n-times the complexity of computing the full gradient, so stochastic gradient methods generally have a much smaller per-iteration complexity than full-gradient methods.Moreover, it has recently been shown that the optimal convergence rates of stochastic gradient methods are O √ n/T 2 without strong convexity and O θ −T S with θ S def = 1 + µ Ln when g is µ-strongly convex, matching the optimal dependence on T and κ of full-gradient methods [33]. 1 Stochastic gradient methods have undergone several revolutions to improve their convergence rates before achieving this lower bound.We summarise these revolutions below, beginning with traditional stochastic gradient descent.

Stochastic Gradient Descent (SGD).
Stochastic gradient descent, dating back to [28], uses the gradients ∇f j , ∀j ∈ J k ⊂ {1, 2, . . ., n} to estimate the full gradient.The mini-batch J k is an index set chosen uniformly at random from all subsets of {1, 2, . . ., n} with cardinality b def = |J k |.When b n, the per-iteration complexity of stochastic gradient descent is much less than full-gradient methods.However, the per-iteration savings come at the cost of a slower convergence rate, as SGD converges at a rate of O(1/ √ T ) in the worst case.Still, SGD outperforms full-gradient methods on many problems, especially if a low-accuracy solution is acceptable.
Variance Reduction.Variance-reduced estimators use gradient information from previous iterates to construct a better estimate of the gradient at the current step, ensuring that the mean-squared error of these estimates decreases as the iterations increase.Variance-reduction improves the convergence rates of stochastic gradient methods, but either have a higher per-iteration complexity or have larger storage requirements than SGD.The two most popular variance-reduced algorithms are SVRG [18] and SAGA [13], which use the following estimators to approximate ∇f (x k+1 ): In SVRG, the full gradient ∇f ( x) is computed every m ≈ 2n iterations, and ∇f ( x) is stored and used for future gradient estimators.SAGA takes a similar approach, storing n past stochastic gradients, and updating the stored gradients so that ∇f j (ϕ j k+1 ) = ∇f j (x k+1 ).In this work, we consider a variant of SVRG where the full gradient is computed at every iteration with probability 1/p ∈ (0, 1] rather than deterministically computing the full gradient every 2n iterations.SVRG, SAGA, and related variance-reduced methods converge at a rate of O (n/T ) when no strong convexity is present.With strong convexity, these algorithms enjoy linear convergence, with a rate of O( 1 + (n + κ) −1 −T ).Both of these rates match the rates of full-gradient methods in terms of their dependence on T and κ.Although these convergence rates are significantly faster than the rate of SGD, they do not achieve the fastest possible dependencies of O 1/T 2 without strong convexity and O((1 + κ −1/2 ) −T ) with strong convexity.
Variance Reduction with Bias.SAGA and SVRG are unbiased gradient estimators because they satisfy E k ∇ k+1 = ∇f (x k+1 ), where E k is the expectation conditioned on the first k iterates.There are several popular variance-reduced algorithms that use biased gradient estimators [23,29].In [14], the authors develop a framework for proving convergence guarantees for biased methods, suggesting that the convergence rates of biased stochastic gradient estimators depend on the sum of two terms: These terms are the mean-squared error (MSE) of the gradient estimator and the "bias term", respectively.The authors also show that recursive gradient estimators such as SARAH [23] and SARGE [14] minimise these terms better than other biased or unbiased estimators, leading to better convergence rates in some settings.The SARAH gradient estimator is As with SVRG, we consider a slight variant of the SARAH estimator in this work, where we compute the full gradient at every step with probability 1/p.The SARGE gradient estimator is similar to the SAGA estimator.
where the variables ψ i k follow the update rule ψ j k+1 = ∇f j (x k+1 )− 1 − b n ∇f j (x k ) for all j ∈ J k , and ψ i k+1 = ψ i k otherwise.Like SAGA, SARGE uses stored gradient information to avoid having to compute the full gradient.These estimators differ from SAGA and SVRG because they are biased (i.e., E k ∇ k+1 = ∇f (x k+1 )).Many works have recently shown that algorithms using the SARAH or SARGE gradient estimators achieve faster convergence rates than algorithms using other estimators in certain settings.Importantly, these recursive gradient methods produce algorithms that achieve the oracle complexity lower bound for non-convex composite optimisation [14,15,26,32,38].They have not yet been shown to achieve optimal convergence rates for convex problems.
Variance Reduction with Negative Momentum.Starting with Katyusha [2] and followed by many others [1,3,4,19,30,36,37], a family of stochastic gradient algorithms have recently emerged that achieve the optimal convergence rates implied by [33].There are two components to these algorithms that make this acceleration possible.First, these algorithms incorporate momentum into each iteration, either through linear coupling [6], as in the case of [1,2,3,4,36], or in a more traditional manner reminiscent of Nesterov's accelerated gradient descent [30,37].Second, these algorithms incorporate an "anchor-point" into their momentum updates that supposedly softens the negative effects of bad gradient evaluations.Almost all of these algorithms are an accelerated form of SVRG with updates of the form using traditional acceleration or linear coupling, respectively (z k and y k are as defined in Algorithm 1, and ).We see that these updates "attract" the current iterate toward a "safe" point, x, where we know the full gradient.Because of this "attractive" rather than "repulsive" quality, updates of this type have been termed "negative momentum".
There are several issues with negative momentum.Most importantly, negative momentum is algorithmspecific.Unlike Nesterov's method of momentum or linear coupling, negative momentum cannot be applied to other stochastic gradient algorithms.SAGA, for example, cannot be accelerated using negative momentum of this form, because there does not exist a point x where we compute the full gradient (however, see [36]).Also, numerical experiments show that negative momentum is often unnecessary to achieve acceleration (see the discussion in [2] or Section 7, for example), suggesting that acceleration is possible without it.

Contributions.
In this work, we show that stochastic gradient algorithms do not require negative momentum to achieve accelerated convergence rates.We introduce the MSEB property, a property that implies Algorithm 1 A Universal Framework for Acceleration Input: Set step size γ k and momentum parameter τ k as in Theorem 5 if µ = 0 or as in Theorem 6 otherwise, and gradient estimator ∇.

5:
end for natural bounds on the bias and MSE of a gradient estimator, and we prove accelerated convergence rates for all MSEB gradient estimators.As special cases, we show that incorporating the SAGA, SVRG, SARAH, and SARGE gradient estimators into the framework of Algorithm 1 creates a stochastic gradient method with an O 1/T 2 convergence rate without strong convexity, and a linear convergence rate that scales with √ κ when strong convexity is present, achieving the optimal convergence rates in both cases up to a constant depending on the bias and MSE of the estimator.
Roadmap.We introduce our algorithm and state our main result in Section 2. We compare our results to existing work in Section 3. The next four sections are devoted to proving our main results.In Section 4, we review elementary results on the subdifferential relation, results on the proximal operator, and lemmas from convex analysis.We prove a general inequality for accelerated stochastic gradient methods using any stochastic gradient estimator in Section 5.This inequality implies that many stochastic gradient methods can be accelerated using our momentum scheme; to prove an accelerated convergence rate for a specific algorithm, we only need to apply an algorithm-specific bound on the MSE and bias of the gradient estimator.We do this for the SAGA, SVRG, SARAH, and SARGE gradient estimators in Section 6.Finally, in Section 7, we demonstrate the performance of our algorithms in numerical experiments.

Algorithm and Main Results
The algorithm we propose is outlined in Algorithm 1. Algorithm 1 takes as input any stochastic gradient estimator ∇ k+1 , so it can be interpreted as a framework for accelerating existing stochastic gradient methods.This algorithm incorporates momentum through linear coupling [6], but is related to Nesterov's accelerated gradient method after rewriting x k+1 as follows: With τ k = 1, there is no momentum, and the momentum becomes more aggressive for smaller τ k .
We show that as long as the MSE and bias of a stochastic gradient estimator satisfies certain bounds and the parameters γ k and τ k are chosen correctly, Algorithm 1 converges at an accelerated rate.There are three principles for choosing γ k and τ k so that Algorithm 1 achieves acceleration.
1.The step size γ k should be small, roughly O (1/n) with the exact dependence on n decreasing with larger MSE and bias of the gradient estimator.
2. On non-strongly convex objectives, the step size should grow sufficiently slowly, so that γ ) decreasing with larger MSE and bias.
3. The momentum should become more aggressive with smaller step sizes, with For strongly convex objectives, γ k and τ k can be kept constant.
For Algorithm 1 to converge, the stochastic gradient estimator must have controlled bias and MSE.Specifically, we require the estimator to satisfy the MSEB property,2 introduced below.Definition 1 For any sequence {x k+1 }, let ∇ k+1 be a stochastic gradient estimator generated from the points and On a high-level, the MSEB property guarantees that the bias and MSE of the gradient estimator decrease sufficiently quickly with k.

Remark 1
In [8], the authors study the convergence of unbiased stochastic gradient methods under first-and second-moment bounds on the gradient estimator.The bounds implied by the MSEB property are similar, but with the crucial difference that they are non-Markovian; we allow our bound on M k to depend on all preceding iterates, not just x k .
In this work, we show that most existing stochastic gradient estimators satisfy the MSEB property, including SAGA, SVRG, SARAH, SARGE, and the full gradient estimator.We list their associated parameters in the following propositions.

Proposition 1
The full gradient estimator ∇ k+1 = ∇f (x k+1 ) satisfies the MSEB property with Proof.The bias and MSE of the full gradient estimator are zero, so it is clear these parameter choices satisfy the bounds in the MSEB property.
Although trivial, Proposition 1 allows us to show that our analysis recovers the accelerated convergence rates of the inertial forward-backward algorithm as a special case.The MSEB property applies to the SAGA and SVRG estimators non-trivially.

Proposition 2
The SAGA gradient estimator (3) satisfies the MSEB property with We prove Proposition 2 in Appendix B. We are able to choose ρ B = 1 for the SAGA and SVRG gradient estimators because they are unbiased, and we can choose M 2 = 0 and ρ F = 1 for these estimators because they admit Markovian bounds on their variance.This is not true for SARAH and SARGE, but these estimators are still compatible with our framework.We prove Propositions 3 and 4 in Appendices C and D, respectively.Proposition 3 Setting p = O(n), the SARAH gradient estimator (4) satisfies the MSEB property with

Proposition 4
The SARGE gradient estimator (5) satisfies the MSEB property with All gradient estimators satisfying the MSEB property can be accelerated using the framework of Algorithm 1, as the following two theorems guarantee.
Theorem 5 (Acceleration Without Strong Convexity) Suppose the stochastic gradient estimator ∇ k+1 satisfies the MSEB(M 1 , M 2 , ρ M , ρ B , ρ F ) property.Define the constants and τ k = 1 cLγ k .After T iterations, Algorithm 1 produces a point y T satisfying the following bound on its suboptimality: where A similar result gives an accelerated linear convergence rate when strong convexity is present.
Remark 2 Although we prove accelerated convergence rates for many popular gradient estimators, the generality of Theorems 5 and 6 allows our results to extend easily to gradient estimators not considered in this work as well.These include, for example, the gradient estimators considered in [17].
Remark 3 With some manipulation, we see that these rates with c = ν = O(n) are similar to the rates proved for Katyusha.In [2], the author shows that in the non-strongly convex case, Katyusha satisfies Recall that Katyusha follows the algorithmic framework of SVRG; S denotes the epoch number, In the strongly convex case, an appropriately adapted version of Katyusha satisfies Similarly, with c = ρ = O(n), Theorem 6 shows that the iterates of Algorithm 1 satisfy , which again matches the rate of Katyusha.Of course, not all stochastic gradient estimators satisfy the bounds necessary to set c = ν = ρ = O(n), so these optimal rates are conditional on being able to construct an "optimal estimator".SAGA, SVRG, SARAH, and SARGE all require c to be slightly larger than O(n).
The proofs of Theorems 5 and 6 use a linear coupling argument adapted from [6], but we use a different adaptation than the one in [2] used to prove convergence rates for Katyusha.To explain the differences between our approach and existing approaches, let us give a high-level description of linear coupling and the generalisation used in [2].
In [6], the authors suggest that gradient descent and mirror descent can be coupled to create an accelerated algorithm.We do not discuss gradient descent and mirror descent in detail (for this, see [6]), but the main idea of linear coupling can be understood from only two bounds arising from these algorithms.For the purpose of this argument, suppose g ≡ 0, so that F ≡ f .Gradient descent with step size η satisfies the following bound on the decrease of the objective (equation (2.1) in [6]): This bound shows that gradient descent is indeed a descent method; it is guaranteed to make progress at each iteration.The iterates of mirror descent using step size γ satisfy a bound on the sub-optimality of each iterate (equation (2.2) in [6]). ∇f While gradient descent is guaranteed to make progress proportional to ∇f (x k ) 2 each iteration, mirror descent potentially introduces an "error" that is proportional to ∇f (x k ) 2 .Linear coupling takes advantage of this duality.Loosely speaking, by combining the sequence of iterates produced by gradient descent with the sequence produced by mirror descent, the guaranteed progress of gradient descent balances the potential error introduced by mirror descent, accelerating convergence.This argument does not immediately hold for stochastic gradient methods.This is because in addition to the norm ∇f (x k ) 2 arising in inequalities ( 6) and ( 7), we also get the MSE of our gradient estimator ∇f (x k ) − ∇f (x k ) 2 as well as a "bias term".In the stochastic setting, analogues of inequalities ( 6) and ( 7) read where the last inequality is Young's, and If the MSE or bias term is too large, the gradient step is no longer a descent step, and the progress does not balance the "error terms" in each of these inequalities, so we cannot expect linear coupling to offer any acceleration.This problem with the MSE and bias term exists for non-accelerated algorithms as well, and all analyses of stochastic gradient methods bound the effect of these terms, but in different ways.Katyusha and other accelerated algorithms in this family incorporate negative momentum to cancel part of the MSE.
In contrast, analyses of non-accelerated algorithms do not try to cancel any of the variance, but show that the variance decreases fast enough so that it does not affect convergence rates.

Related Work
Besides Katyusha, there are many algorithms that use negative momentum for acceleration.In [30], the authors consider an accelerated version of SVRG that combines negative momentum with Nesterov's momentum to achieve the optimal √ κ dependence in the strongly convex case.This approach to acceleration is almost the same as Katyusha, but uses a traditional form of Nesterov's momentum instead of linear coupling.MiG [37] is another variant of these algorithms, corresponding to Katyusha with a certain parameter set to zero.VARAG is another approach to accelerated SVRG using negative momentum.VARAG achieves optimal convergence rates in the non-strongly convex and strongly convex settings under the framework of a single algorithm, and it converges linearly on problems that admit a global error bound, a quality that other algorithms have not yet been shown to possess [19].
The only direct acceleration of a SAGA-like algorithm is SSNM from [36].Using the notation of (3), SSNM chooses a point from the set {ϕ i k } n i=1 uniformly at random, and uses this point as the "anchor point" for negative-momentum acceleration.Although SSNM admits fast convergence rates, there are a few undesirable qualities of this approach.SAGA has heavy storage requirements because it must store n gradients from previous iterations, and SSNM exacerbates this storage problem by storing n points from previous iterations as well.SSNM must also compute two stochastic gradients each iteration, so its per-iteration computational cost is similar to SVRG and Katyusha, and always higher than SAGA's.
Many algorithms for non-convex optimisation also use negative momentum for acceleration.KatyushaX [3] is a version of Katyusha adapted to optimise sum-of-non-convex objectives.To achieve its acceleration, KatyushaX uses classical momentum and a "retraction step", which is effectively an application of negative momentum (this relationship is acknowledged in [3] as well).Natasha [1] and Natasha2 [4] are accelerated algorithms for finding stationary points of non-convex objectives.Both algorithms employ a "retraction step" that is similar to negative momentum [1].
There are also many accelerated stochastic gradient algorithms that do not use negative momentum.In [24], the author applies Nesterov's momentum to SVRG without any sort of negative momentum, proving a linear convergence rate in the strongly convex regime.However, the proven convergence rate is suboptimal, as it implies even worse performance than SVRG when the batch size is small and worse performance than accelerated full-gradient methods when the batch size is close to n.Our results show that a particular application of Nesterov's momentum to SVRG does provide acceleration.
Point-SAGA [12] is another SAGA-like algorithm that achieves optimal convergence rates, but point-SAGA must compute the proximal operator corresponding to F rather than the proximal operator corresponding to g.This is not possible in general, even if the proximal operator corresponding to g is easy to compute, so point-SAGA applies to a different class of functions than the class we consider in this work.
There are also many algorithms that indirectly accelerate stochastic gradient methods.This class of algorithms include Catalyst [20], APPA [16], and the primal-dual methods in [35].These algorithms call a variance-reduced stochastic gradient method as a subroutine, and provide acceleration using an inner-outer loop structure.These algorithms are often difficult to implement in practice due to the difficulty of solving their inner-loop subproblems, and they achieve a convergence rate that is only optimal up to a logarithmic factor.

Preliminaries
In this section, we present some basic definitions and results from optimisation and convex analysis.Much of our analysis involves Bregman divergences.The Bregman divergence associated with a function h is defined as where ξ ∈ ∂h(x) and ∂ is the subdifferential operator.If h is differentiable, we drop the superscript ξ as the subgradient is unique.The function h is convex if and only if D ξ h (y, x) ≥ 0 for all x and y.We say h is µ-strongly convex with µ ≥ 0 if and only if Bregman divergences also arise in the following fundamental inequality.
Lemma 7 ([22], Thm.2.1.5)Suppose f is convex with an L-Lipschitz continuous gradient.We have for all Lemma 7 is equivalent to the following result, which is more specific to our analysis due to the finite-sum structure of the smooth term in (1).
Proof.This follows from applying Lemma 7 to each component f i .
The proximal operator is defined as The proximal operator is also defined implicitly as y − prox g (y) ∈ ∂g(prox g (y)).From this definition of the proximal operator, the following standard inequality is clear.
Lemma 9 Suppose g is µ-strongly convex with µ ≥ 0, and suppose z = prox ηg (x − ηd) for some x, d ∈ R m and constant η.Then, for any Proof.By the strong convexity of g, From the implicit definition of the proximal operator, we know that Multiplying by η and rearranging yields the assertion.

The Acceleration Framework
To apply the linear coupling framework, we must couple stochastic analogues of ( 6) and (7) to construct a lower bound on the one-iteration progress of Algorithm 1.

Lemma 10 (One-Iteration Progress)
The following bound describes the progress made by one iteration of Algorithm 1.
Proof.We use a linear coupling argument.The extrapolated iterate x k+1 can be viewed as a convex combination of an iterate produced from mirror descent (namely, z k ) and one from gradient descent (y k ).This allows us to provide two bounds on the term f (x k+1 ) − f (x * ): one is a regret bound inspired by the classical analysis of mirror descent, and the other is inspired by the traditional descent guarantee of gradient descent.
Inequality 1 uses the convexity of f , 2 follows from the fact that ).We proceed to bound the inner product ∇ k+1 , z k+1 − x * involving the sequence z k+1 using a regret bound from mirror descent, and we bound the term ∇ k+1 , x k+1 − y k+1 using an argument similar to the descent guarantee of gradient descent.By Lemma 9 with z = z k+1 , x = z k , y = x * , d = ∇ k+1 , and η = γ k , For the other term, Inequality 1 follows from the Lipschitz continuity of ∇f i , 2 is Young's inequality, and 3 uses the convexity of g and the update rule Combining inequalities ( 9) and ( 10) with ( 8) and rearranging yields the assertion.
Lemma 10 completes the linear coupling part of our argument.If not for the MSE and bias terms, we could telescope this inequality as in [6] and prove an accelerated convergence rate.As with all analyses of stochastic gradient methods, we need a useful bound on these qualities of the estimator.
Existing analyses of unbiased stochastic gradient methods bound the variance term by a pair of terms that telescope over several iterations, showing that the variance tends to zero with the number of iterations.It is difficult to generalise these arguments to accelerated stochastic methods because one must prove that the variance decreases at an accelerated rate that is inconsistent with existing variance bounds.In the analysis of Katyusha, negative momentum cancels part of the variance term, leaving telescoping terms that decrease at an accelerated rate.Without negative momentum, we must handle the variance term differently.
In the inequality of Lemma 10, we have two non-positive terms: This makes our strategy clear: we must bound the MSE and bias terms by terms of the form x k+1 − y k+1 The bias term can be bounded as Proof.Because z k is independent of the first k − 1 iterates, we can use the MSEB property to say Equality 1 is due to the MSEB property.We are able to pass the conditional expectation into the second inner product in 2 because z k−1 is independent of ∇ k conditioned on the first k − 2 iterates, and inequality 3 is Young's.We can repeat this process once more, applying the MSEB property to obtain Inequality 4 uses our hypothesis on the decrease of s k .This is a recursive inequality, and expanding the recursion yields Taking the sum over the iterations k = 0 to k = T − 1, we apply an estimate to simplify this bound.
Equality 1 is the identity This bound on the bias term includes the MSE, so to complete our bound on the bias term, we must combine Lemma 11 with the following lemma.
Lemma 12 (MSE Bound) Suppose the stochastic gradient estimator ∇ k+1 satisfies the MSEB(M 1 , M 2 , ρ M , ρ B , ρ F ) property, let ρ = min{ρ M , ρ B , ρ F }, and let {s k } be any non-negative sequence satisfying s . The MSE of the gradient estimator is bounded as Proof.First, we derive a bound on the sequence F k arising in the MSEB property.Taking the sum from , and 2 uses the same estimate as in (11).With this bound on F k , we proceed to bound M k in a similar fashion.Lemmas 11 and 12 show that it is possible to cancel the bias term and the MSE using the non-negative terms appearing in the inequality of Lemma 10.Without these terms, we can telescope this inequality over several iterations and prove accelerated convergence rates.We are now prepared to prove Theorems 5 and 6.
Proof of Theorem 5. We set µ = 0 in the inequality of Lemma 10, apply the full expectation operator, and sum the result over the iterations k = 0 to k = T − 1.
We bound the terms in the final line, beginning with the bias term.Our choice for so with s k = γ k and σ k = 1, we apply Lemma 11.This gives where we have dropped the term −1/2E z T − x * 2 because it is non-positive.Applying Lemma 12 to bound the MSE, we have With the parameters set as in the theorem statement, it is clear that the final two lines of ( 12) are non-positive (see Appendix A for a proof).This allows us to drop these lines from the inequality, leaving Rewriting τ k in terms of γ k shows that this is equivalent to , allowing the F (y k ) terms to telescope.Hence, our inequality is equivalent to Using the facts that cLγ 2 , and F (y k ) ≤ F (x * ), we have Thie proves the assertion.
A similar argument proves an accelerated linear convergence rate when strong convexity is present.
Proof of Theorem 6.We recall the inequality of Lemma 10.
By our choice of γ and τ , we have Therefore, we can extract a factor of (1 + µγ) from the left. ( Multiplying this inequality by (1 + µγ) k , summing over iterations k = 0 to k = T − 1, and applying the full expectation operator, we obtain the bound As in the proof of Theorem 5, we bound the bias term and the MSE using Lemmas 11 and 12, respectively.To apply Lemma 11, we let σ k = (1+µγ) k and s k = γ.These choices are appropriate because (1+µγ due to the fact that µγ ≤ ρ/2.Combining these bounds with (13), we have The parameter settings in the theorem statement ensure the final two lines are non-positive (see Appendix A for details).This gives us which is the desired result.

Convergence Rates for Specific Estimators
In light of Theorems 5 and 6, we must only establish suitable bounds on the MSE and bias terms of a gradient estimator to prove accelerated convergence rates for Algorithm 1.We consider four variance-reduced gradient estimators: SAGA, SVRG, SARAH, and SARGE, beginning with the unbiased estimators.We defer proofs to the Appendix.To preserve the generality of our framework, we have not optimised the constants appearing in the presented convergence rates.
Theorem 13 (SAGA Convergence Rates) When using the SAGA gradient estimator in Algorithm 1, set , where where K 2 is defined as in Theorem 6.
It is enlightening to compare these rates to existing convergence rates for full and stochastic gradient methods.In the non-strongly convex setting, our convergence rate is O n 2 /T 2 , matching that of Katyusha [2].In the strongly convex case, if F is poorly conditioned so that L/µ ≥ O(b), we prove linear convergence . With b = n 2/3 , this rate matches the convergence rate of inertial forwardbackward on the same problem (i.e., the rate is independent of n), but we require only n 2/3 stochastic gradient evaluations per iteration compared to the n evaluations that full gradient methods require.This is reminiscent of the results of [5,27], where the authors show that SAGA and SVRG achieve the same convergence rate as full gradient methods on non-convex problems using only n 2/3 stochastic gradient evaluations at each iteration.This is slightly worse than the results proven for Katyusha, which requires O( √ n) stochastic gradient evaluation per iteration to match the convergence rate of full-gradient methods.
The analogous convergence guarantees for SVRG are included in Theorem 14.

Theorem 14 (SVRG Convergence Rates) When using the SVRG gradient estimator in
where where K 2 is defined as in Theorem 6.
The convergence rates for SVRG are similar to the rates for SAGA if p and b are chosen appropriately.In the strongly convex case, setting b = p 2 allows SVRG to match the convergence rate of full gradient methods, and the expected number of stochastic gradient evaluations per iteration is n/p + b.To minimise the number of stochastic gradient evaluations while maintaining the convergence rate of full gradient methods, we set p = O(n 1/3 ), showing that Algorithm 1 using the SVRG gradient estimator achieves the same convergence rate as full gradient methods using only O(n 2/3 ) stochastic gradient evaluations per iteration.

Remark 4
The above discussion shows that when using the SAGA gradient estimator on a strongly convex objective with b = O(n 2/3 ) and iterations.This is compared to the complexity lower bound of O( √ nκ log(1/ )) achieved by Katyusha [33].SVRG has a similar complexity when b = n 2/3 and p = n 1/3 .
The SARAH gradient estimator is similar to the SVRG estimator, as both estimators require the full gradient to be computed periodically.SARAH differs from SVRG by using previous estimates of the gradient to inform future estimates.The recursive nature of the estimator seems to decrease its MSE, which can be observed in experiments and in theory [14,23].However, this comes at the cost of introducing bias into the estimator.
Biased stochastic gradient methods are underdeveloped compared to their unbiased counterparts.The convergence proofs for biased algorithms are traditionally complex and difficult to generalize (see [29], for example), and proximal support has only recently been extended to biased stochastic gradient methods in the convex setting [14].It is difficult to determine conclusively if the negative effect of the bias outweighs the benefits of a lower MSE.We show that Algorithm 1 is able to achieve accelerated rates of convergence using biased estimators as well, but the dependence on n in these complexity bounds is worse than it is for SAGA, SVRG, and the full gradient estimator. where If g is µ-strongly convex, set γ = min and τ = µγ.After T iterations, the point z T satisfies where K 2 is defined as in Theorem 6.
We provide a proof of this result in Appendix C. Theorem 15 shows that using the SARAH gradient estimator in Algorithm 1 achieves an optimal O 1/T 2 convergence rate on convex objectives, but with p = O(n), the constant is a factor of n 2 worse than it is for accelerated SAGA, SVRG, and Katyusha.In the strongly convex case, setting p = O(n) and b = O(1) guarantees a linear convergence rate of O((1 + n −2 µ/L) −T ), achieving the optimal dependence on the condition number, but with a constant that is a factor of n worse than accelerated SAGA and SVRG, and a factor of n3/2 worse than Katyusha.Despite this dependence on n, experimental results, including those in Section 7 and [23], show that the SARAH gradient estimator exhibits competitive performance.
Finally, we provide convergence rates for the SARGE estimator.In [14], the authors introduce the SARGE gradient estimator to mimic the recursive nature of SARAH but trade larger storage costs for a lower average per-iteration complexity, similar to the relationship between SAGA and SVRG.We prove in Appendix D that SARGE satisfies the MSEB property with similar constants to SARAH, and achieves similar convergence rates as well.and τ k = 1 cLγ k .After T iterations, the suboptimalty at y T satisfies where and τ = µγ.After T iterations, the point z T satisfies where K 2 is defined as in Theorem 6.
The convergence rates for SARGE are of the same order as the convergence rates for SARAH, even though SARGE requires fewer stochastic gradient evaluations per iteration on average.

Numerical Experiments
To test our acceleration framework, we use it to accelerate SAGA, SVRG, SARAH, and SARGE on a series of ridge regression and LASSO tasks using the binary classification data sets australian, mushrooms,  phishing, and ijcnn1 from the LIBSVM4 database.We include Katyusha and Katyusha ns for comparison as well.For SVRG and SARAH, we compare our accelerated variants that compute the full gradient probabilistically to the non-accelerated versions that compute the full gradient deterministically at the beginning of each epoch.
With feature vectors a i and labels y i for i ∈ {1, 2, • • • , n}, ridge regression and LASSO can be written as min where R ≡ 1 2 • 2 in ridge regression and R ≡ • 1 for LASSO.Letting g ≡ λR, it is clear that g is λ-strongly convex in ridge regression and g is not strongly convex for LASSO.In all our experiments, we rescale the value of the data to [−1, 1].For ridge regression, we set λ = 1/n, and for LASSO, we set λ = 1/ √ n.For accurate comparisons, we automate all our parameter tuning.For our experiments using ridge regression, we select the step size and momentum parameters from the set {1/t : t ∈ N}.For LASSO, we use the parameters suggested by Theorem 5, but we scale the step size by a constant s ∈ N, and we rescale the momentum parameter so that τ 0 = 1/2.We perform the same parameter-tuning procedure for Katyusha, and set the negative momentum parameter τ 2 = 1/2 as suggested in [2] unless otherwise stated.In our accelerated variants of SVRG and SARAH, we set p = 1 2n , and for the non-accelerated variants and Katyusha, we set the epoch length to 2n.
We measure performance with respect to the suboptimality F (x k+1 ) − F (x * ), where x * is a low-tolerance solution found using forward-backward.To fairly compare algorithms that require a different number of stochastic gradient evaluations per iteration, we report their performance with respect to the number of effective full gradient computations they perform on average each iteration.By this metric, SAGA performs 1/n full gradient computations each iteration, while SVRG performs an average of 2 n + 1 2n , for example.Figures 1 and 2 display the median of 100 trials of ridge regression and LASSO, respectively.We observe the following trends: • Acceleration without negative momentum significantly improves the performance of SAGA, SVRG, SARAH, and SARGE in most cases.The improvement is least dramatic on the smallest data set, australian, and slightly less dramatic for the biased algorithms, SARAH and SARGE.
• Because they require only one stochastic gradient evaluation per iteration, SAGA and Accelerated SAGA require significantly less computation to achieve the same accuracy as other methods.
• In the strongly convex setting, Katyusha performs similarly to or better than SVRG with acceleration in most cases.
• In the non-strongly convex setting, Katyusha ns performs much worse than other methods when using negative momentum.Without negative momentum, it performs much better than all algorithms except Accelerated SAGA.Because Katyusha without negative momentum is almost exactly the same algorithm as Accelerated SVRG, this improved performance is likely due to the second proximal step and additional step size η in Katyusha.All of the algorithms presented in this work can adopt these features without changing their convergence rates.

Conclusion
Although acceleration is a widely used and an extensively researched technique in first-order optimisation, its application to stochastic gradient methods is still poorly understood.The introduction of negative momentum adds another layer of complexity to this line of research.Although algorithms using negative momentum enjoy fast convergence rates and strong performance when the parameters are tuned appropriately, it is unclear if negative momentum is necessary for acceleration.In this work, we propose a universal framework for accelerating stochastic gradient methods that does not rely on negative momentum.Because our approach does not rely on negative momentum, it applies to a much broader class of stochastic gradient estimators.As long as the estimator admits natural bounds on its bias and MSE, it can be used in our framework to produce an accelerated stochastic gradient method with an optimal 1/T 2 dependence on convex problems and an optimal √ κ dependence in the strongly convex setting.The bias and MSE of the estimator appear only in the constants of our convergence rates.From this perspective, negative momentum is effectively a variance-reduction technique, reducing the variance in the iterates to improve the dependence on n in the convergence rates.A natural question for future research is whether there exist gradient estimators with smaller bias and MSE than SAGA, SVRG, SARAH, and SARGE that can be accelerated using our framework and admit a better dependence on n.

A Proofs of Non-Positivity
The goal is to show that the two terms are non-positive with the parameter choices of Theorems 5 and 6.We consider three cases.
Case 1.Let γ k and τ k be as in the statement of Theorem 5.For the first term in (14), ensures that this quadratic in c is non-positive.For the second term, we require τ k ≤ 1/2 for all k, which holds because The constraint c ≥ 16Θ 1 Θ 2 implies that this quantity is non-positive.
Case 2. Let γ and τ be as in the statement of Theorem 6, and suppose which is non-positive due to the constraints on c.For the second term, all we must show is that 1 , and c is larger than 4, so the constraint c ≥ 16Θ 1 Θ 2 ensures that the second term in ( 14) is non-positive.Case 3. In Theorem 6, suppose instead that ρ 2µ ≤ 1 √ µLc , so that γ = ρ 2µ and τ = ρ 2 .This assumption implies the inequality L µ ≤ 4 cρ 2 , so This is a quadratic in c with the root Because c is larger than this quantity, this term is non-positive.For the second term in ( 14), where the last inequality follows from the fact that c ≥ 16Θ 1 Θ 2 .

B Proofs for SAGA and SVRG
Our results for SAGA and SVRG require the following lemma, which appears also as [27,Lem. 7].
Proof.Our hypotheses on these random variables imply E k [X i X j ] = 0 for i = j.Therefore, We begin with a standard bound on the variance ∇ SAGA k+1 − ∇f (x k+1 ) 2 that is an easy consequence of the variance bound in [13], but [13] and related works [2,12,18,34] ultimately use a looser bound in their convergence analysis.

Lemma 18
The variance of the SAGA gradient estimator with minibatches of size b is bounded as follows: Proof.Let X i = ∇f j (x k+1 ) − ∇f j (ϕ j k ) for j ∈ J k .We then have Equality 1 is due to Lemma 17.
Lemma 18 provides a variance bound that is compatible with the MSEB property, as we show in the following lemma.
Proof.Lemma 18 shows that the MSE of the SAGA gradient estimator is dominated by , so we choose this sequence for M k .Using the inequality a − c Equality 1 follows from computing expectations and the update rule for ϕ i k : A similar result holds for the SVRG gradient estimator.

Corollary 20 The SVRG gradient estimator satisfies the MSEB property with
Proof.Following the same argument as in the proof of Lemma 18, we have the bound The factor 1 − 1/p that appears is due to the fact that ∇ k+1 = ∇f (x k+1 ) with probability 1/p.With 2 , we follow the proof of Lemma 19.
Equality 1 follows from the fact that x = x k with probability 1/p.
With the MSEB property established for the SAGA and SVRG gradient estimators, we can apply Theorems 5 and 6 to get a rate of convergence.For the SAGA estimator, Lemma 19 ensures that the choices c =

C Proofs for SARAH
To prove the convergence rates of Theorem 15, we first show that the SARAH gradient estimator satisfies the MSEB property.
Proof.The SARAH gradient estimator is equal to ∇f (x k+1 ) with probability 1/p, so the expectation of the SARAH gradient estimator is Next, we prove a bound on the MSE.Let E k,p denote the expectation conditioned on the first k iterations and the event that the full gradient is not computed at iteration k + 1.Under the condition that the full gradient is not computed, the expectation of the SARAH estimator is The beginning of our proof is similar to the proof of the MSE bound in [23,Lem. 2].
We consider each inner product separately.The first inner product is equal to For the next two inner products, we use the fact that With this equality established, we see that the second inner product is equal to The third inner product can be bounded using a similar procedure.
Altogether, we have For the second term, The inequality is Jensen's.This results in the recursive inequality This provides a bound on the MSE under the condition that the full gradient is not computed at iteration k.If the full gradient is computed, the MSE of the estimator is clearly equal to zero, so applying the full expectation operator yields , it is clear that we can take M 1 = 1, ρ M = 1/p, M 2 = 0, and ρ F = 1.
With these MSEB constants established, convergence rates easily follow from Theorems 5 and 6 with c = 144p 4 and ρ = 1/p.

D Proofs for SARGE
For the proofs in this section, we rewrite the SARGE gradient estimator in terms of the SAGA estimator to make the analysis easier to follow.Define the operator where the variables {ξ i k } n i=1 follow the update rules ξ j k+1 = x k and ξ i k+1 = ξ i k for all i ∈ J k .The SARGE estimator is equal to Before we begin, we require a bound on the MSE of the ξ-SAGA gradient estimator that follows immediately from Lemma 19.

Lemma 22
The MSE of the ξ-SAGA gradient estimator satisfies the following bound: Proof.Following the proof of Lemma 18, Inequality 1 is an application of Lemma 17 and the standard variance decomposition.To continue, we follow the proof of Lemma 19.
Equality 2 follows from computing expectations, and 3 uses the estimate 1 Due to the recursive nature of the SARGE gradient estimator, its MSE depends on the difference between the current estimate and the estimate from the previous iteration.This is true for the recursive SARAH gradient estimate as well, but bounding the quantity 2 is a much more straightforward task than bounding the same quantity for the SARGE estimator.The next lemma provides this bound.

Lemma 23
The SARGE gradient estimator satisfies the following bound: We now bound the first two of these three terms separately.Consider the first term.
Equality 1 is the standard variance decomposition, which states that for any random variable X, The second term can be reduced further by computing the expectation.Let j k be any element of J k .The probability that ∇f j k (ϕ j k k ) = ∇f j k−1 (x k ) is equal to the probability that j k ∈ J k−1 , which is b/n.The probability that ∇f j k (ϕ j k k ) = ∇f j k−2 (x k−1 ) is equal to the probability that j k ∈ J k−1 and j k ∈ J k−2 , which is b/n (1 − b/n).Continuing in this way, This implies that We include the inequality of the second line to simplify later arguments.This completes our bound for the first term of (15).For the second term, we recall Lemma 22.
Combining all of these bounds, we have shown Lemma 23 allows us to take advantage of the recursive structure of our gradient estimate.With this lemma established, we can prove a bound on the MSE.

Lemma 24
The SARGE gradient estimator satisfies the following recursive bound: Proof.The beginning of our proof is similar to the proof of the variance bound for the SARAH gradient estimator in [23,Lem. 2].
We consider each inner product separately.The first inner product is equal to For the next two inner products, we use the fact that With this equality established, we see that the second inner product is equal to The third inner product can be bounded using a similar procedure.
where the inequality is Young's.Altogether and after applying the full expectation operator, we have Finally, we bound the last term on the right using Lemma 23.
Lemma 24 shows that the SARGE gradient estimator satisfies the MSEB property with suitably chosen parameters.
and the choices M 2 = 27+12b n 2 and ρ F = b 2n are clear.To prove the convergence rates of Theorem 16, we simply combine the MSEB constants of Corollary 25 with Theorems 5 and 6.
x S the point where the full gradient was computed at the beginning of epoch S, and P = O(n) is the epoch length.In our notation, S = T /P = O (T /n).Theorem 5 with c = ν = O(n) shows that Algorithm 1 achieves a similar convergence rate of

2 and
D f (y k , x k+1 ).The following two lemmas use the MSEB property to establish bounds of this form.Lemma 11 (Bias Term Bound) Suppose the stochastic gradient estimator ∇ k+1 satisfies the MSEB(M 1 , M 2 , ρ M , ρ B , ρ F ) property, let ρ = min{ρ M , ρ B , ρ F }, and let {σ k } and {s k } be any non-negative sequences satisfying

2
uses the same estimate we applied above, 3 uses the inequality a − c 2 ≤ 2 a − b 2 + 2 b − c 2 , and 4 uses Lemma 7 and the Lipschitz continuity of ∇f i .

Figure 1 :
Figure 1: Performance comparison for solving ridge regression among different algorithms.

and 2
follows from the the inequalities 1 + b 2n 1 − b n ≤ 1 − b 2n and 1 + 2n b ≤ 3n b .This shows that we can take M 1 = 3n b 2 , M 2 = 0, and ρ F = 1.Because the SAGA gradient estimator is unbiased, we can clearly set ρ B = 1, proving the claim.