Skip to main content
SpringerLink
Log in

Complexity bounds for primal-dual methods minimizing the model of objective function

  • Full Length Paper
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

We provide Frank–Wolfe (\(\equiv \) Conditional Gradients) method with a convergence analysis allowing to approach a primal-dual solution of convex optimization problem with composite objective function. Additional properties of complementary part of the objective (strong convexity) significantly accelerate the scheme. We also justify a new variant of this method, which can be seen as a trust-region scheme applying to the linear model of objective function. For this variant, we prove also the rate of convergence for the total variation of linear model of composite objective over the feasible set. Our analysis works also for quadratic model, allowing to justify the global rate of convergence for a new second-order method as applied to a composite objective function. To the best of our knowledge, this is the first trust-region scheme supported by the worst-case complexity analysis both for the functional gap and for the variation of local quadratic model over the feasible set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Performance of CGM as applied to objective function regularized by a norm was studied in [7] and in [20]. The latter paper treats also the case of unbounded domain.

  2. It seems that such line of arguments was used in the first time in Section 7.5 of [9].

References

  1. Conn, A.B., Gould, N.I.M.: Toint, PhL: Trust Region Methods. SIAM, Philadelphia (2000)

    Book  Google Scholar 

  2. Dunn, J.: Convergence rates for conditional gradient sequences generated by implicit step length rules. SIAM J. Control Optim. 18(5), 473–487 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  3. Frank, M., Wolfe, P.: An algorithm for quadratic programming. Nav. Res. Logist. Q. 3, 149–154 (1956)

    Article  MathSciNet  Google Scholar 

  4. Freund, R.M., Grigas, P.: New analysis and results for the Frank-Wolfe method. Math. Program. (2014). doi:10.1007/s10107-014-0841-6

    MATH  Google Scholar 

  5. Garber, D., Hazan, E.: A linearly convergent conditional gradient algorithm with application to online and stochastic optimization (2013). arXiv:1301.4666v5

  6. Garber, D., Hazan, E.: Faster rates for the Frank–Wolfe method over strongly-convex sets (2015). arXiv:1406.1305v2

  7. Harchaoui, Z., Juditsky, A., Nemirovski, A.: Conditional gradient algorithms for norm-regularized smooth convex optimization. Math. Program. (2014). doi:10.1007/s10107-014-0778-9

    MATH  Google Scholar 

  8. Jaggi, M.: Revisiting Frank–Wolfe: projection-free sparse convex optimization. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia (2013)

  9. Juditsky, A., Nemirovski, A.: Solving variational inequalities with monotone operators on domains given by linear minimization oracles. Math. Program. Ser. A 156, 221–256 (2015). doi:10.1007/s10107-015-0876-3

  10. Lacoste-Julien, S., Jaggi, M., Schmidt, M., Pletscher, P.: Block-coordinate Frank–Wolfe optimization of structural svms. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia (2013)

  11. Lan, G.: The complexity of large-scale convex programming under a linear optimization oracle (2014). arXiv:1309.5550v2

  12. Migdalas, A.: A regularization of the Frank-Wolfe method and unification of certain nonlinear programming methods. Math. Program. 63, 331–345 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  13. Nesterov, Y.: Introductory Lectures on Convex Optimization. Kluwer, Boston (2004)

    Book  MATH  Google Scholar 

  14. Nesterov, Y.: Primal-dual subgradient methods for convex problems. Math. Program. 120(1), 221–259 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  15. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  16. Nesterov, Y.: Universal gradient methods for convex optimization problems. Math. Program. (2014). doi:10.1007/s10107-014-0790-0

    MATH  Google Scholar 

  17. Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming. SIAM, Philadelphia (1994)

    Book  MATH  Google Scholar 

  18. Nesterov, Y., Polyak, B.: Cubic regularization of Newton’s method and its global performance. Math. Program. 108(1), 177–205 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  19. Nesterov, Y., Shikhman, V.: Quasi-monotone subgradient methods for nonsmooth convex minimization. JOTA (2014). doi:10.1007/s10957-014-0677-5

    MATH  Google Scholar 

  20. Yu, Y., Zhang, X., Schuurmans, D.: Generalized conditional gradient for sparse estimation (2014). arXiv:1410.4828 (Under review in Journal of Machine Learning Research)

Download references

Acknowledgements

The author is very thankful to both anonymous reviewers for their valuable comments.

Author information

Authors and Affiliations

  1. Center for Operations Research and Econometrics (CORE), Catholic University of Louvain (UCL), 34 voie du Roman Pays, 1348, Louvain-la-Neuve, Belgium

    Yu. Nesterov

Authors
  1. Yu. Nesterov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu. Nesterov.

Additional information

Scientific results of this paper were obtained with support of RSF Grant 17-11-01927.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nesterov, Y. Complexity bounds for primal-dual methods minimizing the model of objective function. Math. Program. 171, 311–330 (2018). https://doi.org/10.1007/s10107-017-1188-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-017-1188-6

Keywords

Mathematics Subject Classification

Search

Navigation