Composite Convex Minimization Involving Self-concordant-Like Cost Functions

  • Quoc Tran-Dinh
  • Yen-Huan Li
  • Volkan Cevher
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 359)

Abstract

The self-concordant-like property of a smooth convex function is a new analytical structure that generalizes the self-concordant notion. While a wide variety of important applications feature the self-concordant-like property, this concept has heretofore remained unexploited in convex optimization. To this end, we develop a variable metric framework of minimizing the sum of a “simple” convex function and a self-concordant-like function. We introduce a new analytic step-size selection procedure and prove that the basic gradient algorithm has improved convergence guarantees as compared to “fast” algorithms that rely on the Lipschitz gradient property. Our numerical tests with real-data sets show that the practice indeed follows the theory.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bach, F.: Self-concordant analysis for logistic regression. Electron. J. Statist. 4, 384–414 (2010)CrossRefMATHMathSciNetGoogle Scholar
  2. 2.
    Bach, F.: Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression (2013)Google Scholar
  3. 3.
    Beck, A., Teboulle, M.: A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems. SIAM J. Imaging Sciences 2(1), 183–202 (2009)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    Becker, S., Candès, E.J., Grant, M.: Templates for convex cone problems with applications to sparse signal recovery. Mathematical Programming Computation 3(3), 165–218 (2011)CrossRefMATHMathSciNetGoogle Scholar
  5. 5.
    Becker, S., Fadili, M.J.: A quasi-Newton proximal splitting method. In: Adv. Neural Information Processing Systems (2012)Google Scholar
  6. 6.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press (2004)Google Scholar
  7. 7.
    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)CrossRefMATHMathSciNetGoogle Scholar
  8. 8.
    Lee, J.D., Sun, Y., Saunders, M.A.: Proximal Newton-type methods for convex optimization. SIAM J. Optim. 24(3), 1420–1443 (2014)CrossRefMATHMathSciNetGoogle Scholar
  9. 9.
    Mine, H., Fukushima, M.: A minimization method for the sum of a convex function and a continuously differentiable function. J. Optim. Theory Appl. 33, 9–23 (1981)CrossRefMATHMathSciNetGoogle Scholar
  10. 10.
    Nesterov, Y.: Introductory lectures on convex optimization: a basic course. Applied Optimization, vol. 87. Kluwer Academic Publishers (2004)Google Scholar
  11. 11.
    Nesterov, Y.: Gradient methods for minimizing composite objective function. Math. Program. 140(1), 125–161 (2013)CrossRefMATHMathSciNetGoogle Scholar
  12. 12.
    Nesterov, Y., Nemirovski, A.: Interior-point Polynomial Algorithms in Convex Programming. Society for Industrial Mathematics (1994)Google Scholar
  13. 13.
    Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer Series in Operations Research and Financial Engineering. Springer (2006)Google Scholar
  14. 14.
    Parikh, N., Boyd, S.: Proximal algorithms. Foundations and Trends in Optimization 1(3), 123–231 (2013)Google Scholar
  15. 15.
    Robinson, S.M.: Strongly Regular Generalized Equations. Mathematics of Operations Research 5(1), 43–62 (1980)CrossRefMATHMathSciNetGoogle Scholar
  16. 16.
    Rockafellar, R.T.: Convex Analysis. Princeton Mathematics Series, vol. 28. Princeton University Press (1970)Google Scholar
  17. 17.
    Schmidt, M., Roux, N.L., Bach, F.: Convergence rates of inexact proximal-gradient methods for convex optimization. In: NIPS, Granada, Spain (2011)Google Scholar
  18. 18.
    Tran-Dinh, Q., Kyrillidis, A., Cevher, V.: Composite self-concordant minimization. J. Mach. Learn. Res. 15, 1–54 (2014) (accepted)Google Scholar
  19. 19.
    Tran-Dinh, Q., Li, Y.-H., Cevher, V.: Composite convex minimization involving self-concordant-like cost functions. LIONS-Tech. Report., 1–19 (2015), http://arxiv.org/abs/1502.01068
  20. 20.
    Yuan, Y.X.: Recent advances in numerical methods for nonlinear equations and nonlinear least squares. Numerical Algebra, Control and Optimization 1(1), 15–34 (2011)CrossRefMATHMathSciNetGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Quoc Tran-Dinh
    • 1
  • Yen-Huan Li
    • 1
  • Volkan Cevher
    • 1
  1. 1.Laboratory for Information and Inference Systems (LIONS)EPFLLausanneSwitzerland

Personalised recommendations