Complexity bounds for primal-dual methods minimizing the model of objective function

Full Length Paper Series A
  • 258 Downloads

Abstract

We provide Frank–Wolfe (\(\equiv \) Conditional Gradients) method with a convergence analysis allowing to approach a primal-dual solution of convex optimization problem with composite objective function. Additional properties of complementary part of the objective (strong convexity) significantly accelerate the scheme. We also justify a new variant of this method, which can be seen as a trust-region scheme applying to the linear model of objective function. For this variant, we prove also the rate of convergence for the total variation of linear model of composite objective over the feasible set. Our analysis works also for quadratic model, allowing to justify the global rate of convergence for a new second-order method as applied to a composite objective function. To the best of our knowledge, this is the first trust-region scheme supported by the worst-case complexity analysis both for the functional gap and for the variation of local quadratic model over the feasible set.

Keywords

Convex optimization Complexity bounds Linear optimization oracle Conditional gradient method Trust-region method 

Mathematics Subject Classification

90-08 90-C25 90-C52 

Notes

Acknowledgements

The author is very thankful to both anonymous reviewers for their valuable comments.

References

  1. 1.
    Conn, A.B., Gould, N.I.M.: Toint, PhL: Trust Region Methods. SIAM, Philadelphia (2000)CrossRefGoogle Scholar
  2. 2.
    Dunn, J.: Convergence rates for conditional gradient sequences generated by implicit step length rules. SIAM J. Control Optim. 18(5), 473–487 (1980)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Frank, M., Wolfe, P.: An algorithm for quadratic programming. Nav. Res. Logist. Q. 3, 149–154 (1956)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Freund, R.M., Grigas, P.: New analysis and results for the Frank-Wolfe method. Math. Program. (2014). doi: 10.1007/s10107-014-0841-6 MATHGoogle Scholar
  5. 5.
    Garber, D., Hazan, E.: A linearly convergent conditional gradient algorithm with application to online and stochastic optimization (2013). arXiv:1301.4666v5
  6. 6.
    Garber, D., Hazan, E.: Faster rates for the Frank–Wolfe method over strongly-convex sets (2015). arXiv:1406.1305v2
  7. 7.
    Harchaoui, Z., Juditsky, A., Nemirovski, A.: Conditional gradient algorithms for norm-regularized smooth convex optimization. Math. Program. (2014). doi: 10.1007/s10107-014-0778-9 MATHGoogle Scholar
  8. 8.
    Jaggi, M.: Revisiting Frank–Wolfe: projection-free sparse convex optimization. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia (2013)Google Scholar
  9. 9.
    Juditsky, A., Nemirovski, A.: Solving variational inequalities with monotone operators on domains given by linear minimization oracles. Math. Program. Ser. A 156, 221–256 (2015). doi: 10.1007/s10107-015-0876-3
  10. 10.
    Lacoste-Julien, S., Jaggi, M., Schmidt, M., Pletscher, P.: Block-coordinate Frank–Wolfe optimization of structural svms. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia (2013)Google Scholar
  11. 11.
    Lan, G.: The complexity of large-scale convex programming under a linear optimization oracle (2014). arXiv:1309.5550v2
  12. 12.
    Migdalas, A.: A regularization of the Frank-Wolfe method and unification of certain nonlinear programming methods. Math. Program. 63, 331–345 (1994)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Nesterov, Y.: Introductory Lectures on Convex Optimization. Kluwer, Boston (2004)CrossRefMATHGoogle Scholar
  14. 14.
    Nesterov, Y.: Primal-dual subgradient methods for convex problems. Math. Program. 120(1), 221–259 (2009)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Nesterov, Y.: Universal gradient methods for convex optimization problems. Math. Program. (2014). doi: 10.1007/s10107-014-0790-0 MATHGoogle Scholar
  17. 17.
    Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming. SIAM, Philadelphia (1994)CrossRefMATHGoogle Scholar
  18. 18.
    Nesterov, Y., Polyak, B.: Cubic regularization of Newton’s method and its global performance. Math. Program. 108(1), 177–205 (2006)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Nesterov, Y., Shikhman, V.: Quasi-monotone subgradient methods for nonsmooth convex minimization. JOTA (2014). doi: 10.1007/s10957-014-0677-5 MATHGoogle Scholar
  20. 20.
    Yu, Y., Zhang, X., Schuurmans, D.: Generalized conditional gradient for sparse estimation (2014). arXiv:1410.4828 (Under review in Journal of Machine Learning Research)

Copyright information

© Springer-Verlag GmbH Germany and Mathematical Optimization Society 2017

Authors and Affiliations

  1. 1.Center for Operations Research and Econometrics (CORE)Catholic University of Louvain (UCL)Louvain-la-NeuveBelgium

Personalised recommendations