Skip to main content
SpringerLink
Log in

On the Convergence Analysis of the Optimized Gradient Method

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

This paper considers the problem of unconstrained minimization of smooth convex functions having Lipschitz continuous gradients with known Lipschitz constant. We recently proposed the optimized gradient method for this problem and showed that it has a worst-case convergence bound for the cost function decrease that is twice as small as that of Nesterov’s fast gradient method, yet has a similarly efficient practical implementation. Drori showed recently that the optimized gradient method has optimal complexity for the cost function decrease over the general class of first-order methods. This optimality makes it important to study fully the convergence properties of the optimized gradient method. The previous worst-case convergence bound for the optimized gradient method was derived for only the last iterate of a secondary sequence. This paper provides an analytic convergence bound for the primary sequence generated by the optimized gradient method. We then discuss additional convergence properties of the optimized gradient method, including the interesting fact that the optimized gradient method has two types of worst-case functions: a piecewise affine-quadratic function and a quadratic function. These results help complete the theory of an optimal first-order method for smooth convex minimization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Nes13 was developed originally to deal with nonsmooth composite convex functions with a line-search scheme [10, Section 4], whereas the algorithm shown here is a simplified version of [10, Section 4] for unconstrained smooth convex minimization (M) without a line-search.

References

  1. Kim, D., Fessler, J.A.: Optimized first-order methods for smooth convex minimization. Math. Program. 159(1), 81–107 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  2. Drori, Y., Teboulle, M.: Performance of first-order methods for smooth convex minimization: A novel approach. Math. Program. 145(1–2), 451–82 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  3. Nesterov, Y.: A method for unconstrained convex minimization problem with the rate of convergence \(O(1/k^2)\). Dokl. Akad. Nauk. USSR 269(3), 543–7 (1983)

    MathSciNet  Google Scholar 

  4. Drori, Y.: The exact information-based complexity of smooth convex minimization (2016). arXiv:1606.01424

  5. Cevher, V., Becker, S., Schmidt, M.: Convex optimization for big data: scalable, randomized, and parallel algorithms for big data analytics. IEEE Sig. Proc. Mag. 31(5), 32–43 (2014)

    Article  Google Scholar 

  6. Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comp. Math. Math. Phys. 4(5), 1–17 (1964)

    Article  Google Scholar 

  7. Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  8. Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Dordrecht (2004)

    Book  MATH  Google Scholar 

  9. Taylor, A.B., Hendrickx, J.M., Glineur, F.: Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Math. Program. (2016). doi:10.1007/s10107-016-1009-3

  10. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  11. Drori, Y., Teboulle, M.: An optimal variant of Kelley’s cutting-plane method. Math. Program. (2016). doi:10.1007/s10107-016-0985-7

  12. Drori, Y.: Contributions to the complexity analysis of optimization algorithms. Ph.D. thesis, Tel-Aviv Univ., Israel (2014)

  13. Taylor, A.B., Hendrickx, J.M., Glineur, F.: Exact worst-case performance of first-order algorithms for composite convex optimization (2015). arXiv:1512.07516

  14. Lessard, L., Recht, B., Packard, A.: Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J. Optim. 26(1), 57–95 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  15. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

This research was supported in part by NIH grant U01 EB018753.

Author information

Authors and Affiliations

  1. University of Michigan, Ann Arbor, MI, USA

    Donghwan Kim & Jeffrey A. Fessler

Authors
  1. Donghwan Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeffrey A. Fessler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Donghwan Kim.

Additional information

Communicated by Jan Sokolowski.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, D., Fessler, J.A. On the Convergence Analysis of the Optimized Gradient Method. J Optim Theory Appl 172, 187–205 (2017). https://doi.org/10.1007/s10957-016-1018-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-016-1018-7

Keywords

Mathematics Subject Classification

Search

Navigation