Mathematical Programming

, Volume 104, Issue 2–3, pp 609–633 | Cite as

Newton methods for nonsmooth convex minimization: connections among Open image in new window-Lagrangian, Riemannian Newton and SQP methods

Article

Abstract

This paper studies Newton-type methods for minimization of partly smooth convex functions. Sequential Newton methods are provided using local parameterizations obtained from Open image in new window-Lagrangian theory and from Riemannian geometry. The Hessian based on the Open image in new window-Lagrangian depends on the selection of a dual parameter g; by revealing the connection to Riemannian geometry, a natural choice of g emerges for which the two Newton directions coincide. This choice of g is also shown to be related to the least-squares multiplier estimate from a sequential quadratic programming (SQP) approach, and with this multiplier, SQP gives the same search direction as the Newton methods.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bonnans, J., Gilbert, J., Lemaréchal, C., Sagastizábal, C.: Numerical Optimization. Universitext. Springer-Verlag, 2003Google Scholar
  2. 2.
    Chamberlain, R., Lemaréchal, C., Pedersen, H., Powell, M.: The watchdog technique for forcing convergence in algorithms for constrained optimization. Math. Program. Study Series 16, 1–17 (1982)Google Scholar
  3. 3.
    Dedieu, J.-P., Priouret, P., Malajovich, G.: Newton's method on Riemannian manifolds: covariant alpha-theory. IMA J. Numer. Anal. 23 (3), 395–419 (2003)CrossRefGoogle Scholar
  4. 4.
    Dennis, J.E., Moré, J.J.: Quasi-Newton methods, motivation and theory. SIAM Review 19(1), 46–89 (1977)CrossRefGoogle Scholar
  5. 5.
    do Carmo, M.P.: Riemannian Geometry. Mathematics: Theory and Applications. Birkhäuser, 1992Google Scholar
  6. 6.
    Edelman, A., Arias, T., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20 (2), 303–353 (1999)CrossRefGoogle Scholar
  7. 7.
    Gabay, D.: Minimizing a differentiable function over a differentiable manifold. J. Optimization Theory Appl. 37(2), 177–219 June 1982CrossRefGoogle Scholar
  8. 8.
    Hare, W.: Recent functions and sets of smooth substructure: Relationships and examples. To appear in J. Comput. Optim. Appl. available at http://www.cecm.sfu.ca/~whare/Subsmooth-relations.ps, Mar. 17, 2005
  9. 9.
    Hiriart-Urruty, J.-B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms. Number 305–306 in Grundlehren der mathematischen Wissenschaften. Springer-Verlag, Berlin, 1993Google Scholar
  10. 10.
    Lee, J.M.: Riemannian Manifolds: An Introduction to Curvature. Number 176 in Graduate Texts in Mathematics. Springer-Verlag, New York, 1997Google Scholar
  11. 11.
    Lemaréchal, C., Oustry, F., Sagastizábal, C.: The Open image in new window-Lagrangian of a convex function. Trans. AMS 352(2), 711–729 (1999)CrossRefGoogle Scholar
  12. 12.
    Lewis, A.S.: Active sets, nonsmoothness and sensitivity. SIAM J. Optim. 13, 702–725 (2003)CrossRefGoogle Scholar
  13. 13.
    Maratos, N.: Exact penalty function algorithms for finite dimensional and control optimization problems. PhD thesis, Imperial College, London, 1978Google Scholar
  14. 14.
    Mayne, D.Q.: On the use of exact penalty functions to determine step length in optimization algorithms. In: Lecture Notes in Mathematics, volume 773, Springer Verlag, 1980 pp 98–109Google Scholar
  15. 15.
    Mifflin, R., Sagastizábal, C.: Open image in new window-decomposition derivatives for convex max-functions. In: M. Théra and R. Tichatschke, editors, Ill-Posed Variational Problems and Regularization Techniques, Springer-Verlag, Berlin, 1999 pp 167–186Google Scholar
  16. 16.
    Mifflin, R., Sagastizábal C.: On Open image in new window-theory for functions with primal-dual gradient structure. SIAM J. Optim. 11 (2), 547–571 (2000)CrossRefGoogle Scholar
  17. 17.
    Mifflin, R., Sagastizábal, C.: Proximal points are on the fast track. J. Convex Anal. 9 (2), 563–579 (2002)Google Scholar
  18. 18.
    Mifflin, R., Sagastizábal, C.: Primal-dual gradient structured functions: second-order results; links to epi-derivatives and partly smooth functions. SIAM J. Optim. 13 (4), 1174–1194 (2003)CrossRefGoogle Scholar
  19. 19.
    Mifflin, R., Sagastizábal, C.: Open image in new window-smoothness and proximal point results for some nonconvex functions. Optim. Methods Softw. 19 (5), 463–478 (2004)CrossRefGoogle Scholar
  20. 20.
    Mifflin, R., Sagastizábal, C.: A Open image in new window-algorithm for convex minimization. Preprint, available at http://www.sci.wsu.edu/math/faculty/mifflin/vualgo.ps, revised Feb. 14, 2005
  21. 21.
    Oustry, F.: The Open image in new window-Lagrangian of the maximum eigenvalue function. SIAM J. Optim. 9 (2), 526–549 (1999)CrossRefGoogle Scholar
  22. 22.
    Overton, M.L.: Large-scale optimization of eigenvalues. SIAM J. Optim. 2, 88–120 (1992)CrossRefGoogle Scholar
  23. 23.
    Overton, M.L., Ye, X.: Towards second-order methods for structured nonsmooth optimization. In: S. Gomez and J. P. Hennart, editors, Advances in Optimization and Numerical Analysis, Kluwer, 1994 pp 97–110Google Scholar
  24. 24.
    Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Number 317 in Grundlehren der mathematischen Wissenschaften. Springer-Verlag, Berlin, 1998Google Scholar
  25. 25.
    Shapiro, A., Fan, M.K.H.: On eigenvalue optimization. SIAM J. Optim. 5 (3), 552–569 (1995)CrossRefGoogle Scholar
  26. 26.
    Smith, S.T.: Optimization techniques on Riemannian manifolds. Fields Inst. Comm. 3, 113–136 (1994)Google Scholar
  27. 27.
    Udrişte, C.: Convex functions and optimization methods on Riemannian manifolds. Number 297 in Mathematics and its Applications. Kluwer, 1994Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  1. 1.Numerica Corp.Ft. CollinsUSA
  2. 2.INRIASaint Ismier CedexFrance

Personalised recommendations