Skip to main content
SpringerLink
Log in

Derivative-Free Optimization Via Proximal Point Methods

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

Derivative-Free Optimization (DFO) examines the challenge of minimizing (or maximizing) a function without explicit use of derivative information. Many standard techniques in DFO are based on using model functions to approximate the objective function, and then applying classic optimization methods to the model function. For example, the details behind adapting steepest descent, conjugate gradient, and quasi-Newton methods to DFO have been studied in this manner. In this paper we demonstrate that the proximal point method can also be adapted to DFO. To that end, we provide a derivative-free proximal point (DFPP) method and prove convergence of the method in a general sense. In particular, we give conditions under which the gradient values of the iterates converge to 0, and conditions under which an iterate corresponds to a stationary point of the objective function.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Note that, although the model function is linear, the computation of the proximal point still contains the quadratic penalty term. If the piecewise linear model function is given by \(f^{k}(x) =\displaystyle\max_{i=1, 2, \ldots ,N} \{ \langle a_i, x \rangle+ b_i \}\), then the resulting subproblem takes the form

    $$\operatorname{argmin}_{v,y} \biggl\{ v + r \frac{1}{2}\|x-y \|^2 :\ v \geq \langle a_i, x \rangle+ b_i \mbox{ for } i=1, 2, \ldots ,N \biggr\}. $$

    (The proximal point of f k is given by y.)

References

  1. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS/SIAM Series on Optimization., vol. 8. SIAM, Philadelphia (2009)

    Book  MATH  Google Scholar 

  2. Booker, A.J., Dennis, J.E. Jr., Frank, P.D., Serafini, D.B., Torczon, V.: Optimization using surrogate objectives on a helicopter test example. In: Computational Methods for Optimal Design and Control, Arlington, VA, 1997. Progr. Systems Control Theory, vol. 24, pp. 49–58. Birkhäuser, Boston (1998)

    Chapter  Google Scholar 

  3. Duvigneau, R., Visonneau, M.: Hydrodynamic design using a derivative-free method. Struct. Multidiscip. Optim. 28, 195–205 (2004)

    Article  Google Scholar 

  4. Marsden, A.L., Wang, M., Dennis, J.E. Jr., Moin, P.: Trailing-edge noise reduction using derivative-free optimization and large-eddy simulation. J. Fluid Mech. 572, 13–36 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  5. Marsden, A.L., Feinstein, J.A., Taylor, C.A.: A computational framework for derivative-free optimization of cardiovascular geometries. Comput. Methods Appl. Mech. Eng. 197(21–24), 1890–1905 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  6. Hare, W.L.: Using derivative free optimization for constrained parameter selection in a home and community care forecasting model. In: International Perspectives on Operations Research and Health Care. Proceedings of the 34th Meeting of the EURO Working Group on Operational Research Applied to Health Sciences, pp. 61–73 (2010)

    Google Scholar 

  7. Audet, C., Dennis, J.E. Jr., Le Digabel, S.: Globalization strategies for mesh adaptive direct search. Comput. Optim. Appl. 46(2), 193–215 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  8. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, New York (2006)

    MATH  Google Scholar 

  9. Coope, I.D., Price, C.J.: A direct search frame-based conjugate gradients method. J. Comput. Math. 22(4), 489–500 (2004)

    MATH  MathSciNet  Google Scholar 

  10. Cheng, W., Xiao, Y., Hu, Q.J.: A family of derivative-free conjugate gradient methods for large-scale nonlinear systems of equations. J. Comput. Appl. Math. 224(1), 11–19 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  11. Martinet, B.: Détermination approchée d’un point fixe d’une application pseudo-contractante. Cas de l’application prox. C. R. Math. Acad. Sci. Paris, Sér. A–B 274, A163–A165 (1972)

    MathSciNet  Google Scholar 

  12. Lemaréchal, C., Strodiot, J.J., Bihain, A.: On a bundle algorithm for nonsmooth optimization. In: Nonlinear Programming, vol. 4, pp. 245–282. Academic Press, New York (1981)

    Google Scholar 

  13. Mifflin, R.: A modification and extension of Lemarechal’s algorithm for nonsmooth minimization. Math. Program. Stud. 17, 77–90 (1982)

    Article  MATH  MathSciNet  Google Scholar 

  14. Lukšan, L., Vlček, J.: A bundle-Newton method for nonsmooth unconstrained minimization. Math. Program., Ser. A 83(3), 373–391 (1998)

    MATH  Google Scholar 

  15. Hare, W., Sagastizábal, C.: Computing proximal points of nonconvex functions. Math. Program., Ser. B 116(1–2), 221–258 (2009)

    Article  MATH  Google Scholar 

  16. Hare, W., Sagastizábal, C.: Redistributed proximal bundle method for nonconvex optimization. SIAM J. Optim. 20(5), 2442–2473 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  17. Mifflin, R., Sagastizábal, C.: Proximal points are on the fast track. J. Convex Anal. 9(2), 563–579 (2002)

    MATH  MathSciNet  Google Scholar 

  18. Hare, W.L., Lewis, A.S.: Identifying active constraints via partial smoothness and prox-regularity. J. Convex Anal. 11(2), 251–266 (2004)

    MATH  MathSciNet  Google Scholar 

  19. Hare, W.L.: A proximal method for identifying active manifolds. Comput. Optim. Appl. 43(2), 295–306 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  20. Mifflin, R., Sagastizábal, C.: \(\mathcal{V}\mathcal{U}\)-smoothness and proximal point results for some nonconvex functions. Optim. Methods Softw. 19(5), 463–478 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  21. Mifflin, R., Sagastizábal, C.: A VU-algorithm for convex minimization. Math. Program., Ser. B 104(2–3), 583–608 (2005)

    Article  MATH  Google Scholar 

  22. Hare, W.L., Poliquin, R.A.: Prox-regularity and stability of the proximal mapping. J. Convex Anal. 14(3), 589–606 (2007)

    MATH  MathSciNet  Google Scholar 

  23. Kiwiel, K.C.: A proximal bundle method with approximate subgradient linearizations. SIAM J. Optim. 16(4), 1007–1023 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  24. Shen, J., Xia, Z.Q., Pang, L.P.: A proximal bundle method with inexact data for convex nondifferentiable minimization. Nonlinear Anal. 66(9), 2016–2027 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  25. Kiwiel, K.C., Lemaréchal, C.: An inexact bundle variant suited to column generation. Math. Program., Ser. A 118(1), 177–206 (2009)

    Article  MATH  Google Scholar 

  26. Kiwiel, K.C.: An inexact bundle approach to cutting-stock problems. INFORMS J. Comput. 22(1), 131–143 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  27. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 317. Springer, Berlin (1998)

    Book  MATH  Google Scholar 

  28. Hiriart-Urruty, J.-B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 305–306. Springer, Berlin (1993)

    Google Scholar 

  29. Mäkelä, M.M.: Survey of bundle methods for nonsmooth optimization. Optim. Methods Softw. 17(1), 1–29 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  30. Conn, A.R., Scheinberg, K., Vicente, L.N.: Geometry of interpolation sets in derivative free optimization. Math. Program., Ser. B 111(1–2), 141–172 (2008)

    MATH  MathSciNet  Google Scholar 

  31. Conn, A.R., Scheinberg, K., Vicente, L.N.: Geometry of sample sets in derivative-free optimization: polynomial regression and underdetermined interpolation. IMA J. Numer. Anal. 28(4), 721–748 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  32. Apkarian, P., Noll, D., Prot, O.: A proximity control algorithm to minimize nonsmooth and nonconvex semi-infinite maximum eigenvalue functions. J. Convex Anal. 16(3–4), 641–666 (2009)

    MATH  MathSciNet  Google Scholar 

Download references

Acknowledgements

Work in this paper was supported by NSERC Discovery grants (Hare and Lucet), a UBC IRF grant (Hare), and a Canadian Foundation for Innovation (CFI) Leaders Opportunity Fund (Lucet) programs.

Author information

Authors and Affiliations

  1. University of British Columbia, Okanagan Campus (UBCO), 3333 University Way, Kelowna, BC, V1V 1V7, Canada

    W. L. Hare

  2. Computer Science, UBCO, Kelowna, Kanada

    Y. Lucet

Authors
  1. W. L. Hare
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Y. Lucet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to W. L. Hare.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hare, W.L., Lucet, Y. Derivative-Free Optimization Via Proximal Point Methods. J Optim Theory Appl 160, 204–220 (2014). https://doi.org/10.1007/s10957-013-0354-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-013-0354-0

Keywords

Search

Navigation