Foundations of Computational Mathematics

, Volume 15, Issue 3, pp 715–732

Adaptive Restart for Accelerated Gradient Schemes

Article

DOI: 10.1007/s10208-013-9150-3

Cite this article as:
O’Donoghue, B. & Candès, E. Found Comput Math (2015) 15: 715. doi:10.1007/s10208-013-9150-3

Abstract

In this paper we introduce a simple heuristic adaptive restart technique that can dramatically improve the convergence rate of accelerated gradient schemes. The analysis of the technique relies on the observation that these schemes exhibit two modes of behavior depending on how much momentum is applied at each iteration. In what we refer to as the ‘high momentum’ regime the iterates generated by an accelerated gradient scheme exhibit a periodic behavior, where the period is proportional to the square root of the local condition number of the objective function. Separately, it is known that the optimal restart interval is proportional to this same quantity. This suggests a restart technique whereby we reset the momentum whenever we observe periodic behavior. We provide a heuristic analysis that suggests that in many cases adaptively restarting allows us to recover the optimal rate of convergence with no prior knowledge of function parameters.

Keywords

Convex optimization First order methods Accelerated gradient schemes 

Mathematics Subject Classification

80M50 90C06 90C25 

Copyright information

© SFoCM 2013

Authors and Affiliations

  1. 1.Stanford UniversityStanfordUSA

Personalised recommendations