Skip to main content
SpringerLink
Log in

Sobolev seminorm of quadratic functions with applications to derivative-free optimization

  • Full Length Paper
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

This paper studies the \(H^1\) Sobolev seminorm of quadratic functions. The research is motivated by the least-norm interpolation that is widely used in derivative-free optimization. We express the \(H^1\) seminorm of a quadratic function explicitly in terms of the Hessian and the gradient when the underlying domain is a ball. The seminorm gives new insights into least-norm interpolation. It clarifies the analytical and geometrical meaning of the objective function in least-norm interpolation. We employ the seminorm to study the extended symmetric Broyden update proposed by Powell. Numerical results show that the new thoery helps improve the performance of the update. Apart from the theoretical results, we propose a new method of comparing derivative-free solvers, which is more convincing than merely counting the numbers of function evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Notice that \(f\) is not always equal to the objective function \(F\), which is the case in Powell [37–41]. See Sect. 2 for details.

  2. We define the convergence on \(\mathcal{Q }\) to be the convergence of coefficients.

  3. Since the trust-region radii of the subsequent iterations vary dynamically according to the degree of success, there should be more elaborate ways to define the domain \(\fancyscript{B}\) adaptively. The definition given here might be the simplest one, and is enough for our experiment.

References

  1. Adams, R.A.: Sobolev Spaces. Pure and Applied Mathematics. Academic Press, London (1975)

    Google Scholar 

  2. Bandeira, A.S., Scheinberg, K., Vicente, L.N.: Computation of sparse low degree interpolating polynomials and their application to derivative-free optimization. Math. Program. 134(1), 223–257 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  3. Booker, A.J., Dennis, J.E., Frank, P.D., Serafini, D.B., Torczon, V., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscip. Optim. 17(1), 1–13 (1999)

    Article  Google Scholar 

  4. Booker, A.J., Dennis Jr, J.E., Frank, P., Serafini, D.B., Torczon, V., Trosset, M.W.: Optimization using surrogate objectives on a helicopter test example. Prog. Syst. Control Theory 24, 49–58 (1998)

  5. Booker, A.J., Frank, P., Dennis Jr, J.E., Moore, D.W., Serafini, D.B.: Managing surrogate objectives to optimize helicopter rotor design-further experiments. In: Proceedings of the Seventh AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization (1998)

  6. Brent, R.P.: Algorithms for Minimization Without Derivatives. Prentice-Hall, Englewood Cliffs (1973)

    MATH  Google Scholar 

  7. Choi, T.D., Kelley, C.T.: Superlinear convergence and implicit filtering. SIAM J. Optim. 10(4), 1149–1162 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  8. Conn, A.R., Gould, N.I.M., Toint, Ph.L.: Trust-Region Methods, MPS-SIAM series on optimization, Vol. 1. Society for Industrial Mathematics (2000)

  9. Conn, A.R., Scheinberg, K., Toint, Ph.L.: On the convergence of derivative-free methods for unconstrained optimization. In: Buhmann, M.D., Iserles, A. (eds.) Approximation Theory and Optimization: Tributes to M. J. D. Powell, pp. 83–108. Cambridge University Press, Cambridge (1997)

  10. Conn, A.R., Scheinberg, K., Toint, Ph.L.: Recent progress in unconstrained nonlinear optimization without derivatives. Math. Program. 79, 397–414 (1997)

    Google Scholar 

  11. Conn, A.R., Scheinberg, K., Toint, Ph.L.: A derivative free optimization algorithm in practice. In: Proceedings of 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization. St. Louis, MO (1998)

  12. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. Society for Industrial and Applied Mathematics, Philadelphia (2009)

    Book  MATH  Google Scholar 

  13. Conn, A.R., Toint, Ph.L.: An algorithm using quadratic interpolation for unconstrained derivative free optimization. In: Di Pillo, G., Giannessi, F. (eds.) Nonlinear Optimization and Applications, pp. 27–47. Kluwer Academic/Plenum Publishers, New York (1996)

  14. Custódio, A.L., Rocha, H., Vicente, L.N.: Incorporating minimum frobenius norm models in direct search. Comput. Optim. Appl. 46(2), 265–278 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  15. Dennis Jr, J.E., Schnabel, R.B.: Least change secant updates for quasi-newton methods. SIAM Rev. 21(4), 443–459 (1979)

  16. Diniz-Ehrhardt, M.A., Martínez, J.M., Raydán, M.: A derivative-free nonmonotone line-search technique for unconstrained optimization. J. Comput. Appl. Math. 219(2), 383–397 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  17. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  18. Evans, L.C.: Partial Differential Equations. Graduate Studies in Mathematics. American Mathematical Society, Providence (1998)

    Google Scholar 

  19. Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, New York (1987)

    MATH  Google Scholar 

  20. Fowler, K.R., Reese, J.P., Kees, C.E., Kelley, C.T., Miller, C.T., Audet, C., Booker, A.J., Couture, G., Darwin, R.W.: Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Res. 31(5), 743–757 (2008)

    Article  Google Scholar 

  21. Gilmore, P., Kelley, C.T.: An implicit filtering algorithm for optimization of functions with many local minima. SIAM J. Optim. 5, 269 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  22. Gould, N.I.M., Orban, D., Toint, PhL: CUTEr and SifDec: a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. (TOMS) 29(4), 373–394 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  23. Hemker, T., Fowler, K., Von Stryk, O.: Derivative-free optimization methods for handling fixed costs in optimal groundwater remediation design. In: Proceedings of the CMWR XVI-Computational Methods in Water Resources, pp. 19–22 (2006)

  24. Kelley, C.: Iterative Methods for Optimization, Frontiers in Applied Mathematics, Vol. 18. Society for Industrial Mathematics (1999)

  25. Kelley, C.T.: A brief introduction to implicit filtering. North Carolina State University, Raleigh, NC, CRSC, Technical Report CRSC-TR02-28 (2002)

  26. Kelley, C.T.: Implicit Filtering. Society for Industrial and Applied Mathematics (2011)

  27. Kolda, T.G., Lewis, R.M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45(3), 385–482 (2003)

    Google Scholar 

  28. Lewis, R.M., Torczon, V., Trosset, M.W.: Direct search methods: then and now. J. Comput. Appl. Math. 124(1), 191–207 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  29. Marazzi, M., Nocedal, J.: Wedge trust region methods for derivative free optimization. Math. Program. 91(2), 289–305 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  30. Oeuvray, R.: Trust-region methods based on radial basis functions with application to biomedical imaging. Ph.D. thesis, École Polytechnique Fédérale de Lausanne (2005)

  31. Oeuvray, R., Bierlaire, M.: A new derivative-free algorithm for the medical image registration problem. Int. J. Model. Simul. 27(2), 115–124 (2007)

    Google Scholar 

  32. Oeuvray, R., Bierlaire, M.: BOOSTERS: A derivative-free algorithm based on radial basis functions. Int. J. Model. Simul. 29(1), 26–36 (2009)

    Google Scholar 

  33. Powell, M.J.D.: A direct search optimization method that models the objective and constraint functions by linear interpolation. In: Gomez, S., Hennart, J.P. (eds.) Advances in Optimization and Numerical Analysis: Proceedings of the Sixth Workshop on Optimization and Numerical Analysis (Oaxaca, Mexico), pp. 51–67. Kluwer Academic, Dordrecht (1994)

  34. Powell, M.J.D.: Direct search algorithms for optimization calculations. Acta Numerica 7(1), 287–336 (1998)

    Article  Google Scholar 

  35. Powell, M.J.D.: UOBYQA: unconstrained optimization by quadratic approximation. Technical Report in DAMTP NA2000/14, CMS, University of Cambridge (2000)

  36. Powell, M.J.D.: On trust region methods for unconstrained minimization without derivatives. Math. Program. 97(3), 605–623 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  37. Powell, M.J.D.: Least frobenius norm updating of quadratic models that satisfy interpolation conditions. Math. Program. 100, 183–215 (2004)

    MATH  MathSciNet  Google Scholar 

  38. Powell, M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. Technical Report in DAMTP NA2004/08, CMS, University of Cambridge (2004)

  39. Powell, M.J.D.: Developments of NEWUOA for minimization without derivatives. IMA J. Numer. Anal. 649–664 (2008) doi:10.1093/imanum/drm047

  40. Powell, M.J.D.: The BOBYQA algorithm for bound constrained optimization without derivatives. Technical Report in DAMTP 2009/NA06, CMS, University of Cambridge (2009)

  41. Powell, M.J.D.: Beyond symmetric Broyden for updating quadratic models in minimization without derivatives. Math. Program. 1–26 (2012) doi:10.1007/s10107-011-0510-y

  42. Stewart III, G.: A modification of davidon’s minimization method to accept difference approximations of derivatives. J. ACM (JACM) 14(1), 72–83 (1967)

    Article  MATH  Google Scholar 

  43. Berghen Vanden, F., Bersini, H.: CONDOR, a new parallel, constrained extension of powell’s UOBYQA algorithm: Experimental results and comparison with the DFO algorithm. J. Comput. Appl. Math. 181(1), 157–175 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  44. Wild, S.M.: MNH: a derivative-free optimization algorithm using minimal norm Hessians. In: Tenth Copper Mountain Conference on Iterative Methods (2008)

  45. Wild, S.M., Regis, R.G., Shoemaker, C.A.: ORBIT: optimization by radial basis function interpolation in trust-regions. SIAM J. Sci. Comput. 30(6), 3197–3219 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  46. Winfield, D.H.: Function minimization by interpolation in a data table. IMA J. Appl. Math. 12(3), 339 (1973)

    Article  MATH  MathSciNet  Google Scholar 

  47. Wright, M.H.: Direct search methods: Once scorned, now respectable. Pitman Research Notes in Mathematics Series, pp. 191–208 (1996)

Download references

Acknowledgments

This work is based on sections 4.3–4.5 of my PhD thesis, which was written under the supervision of Professor Ya-xiang Yuan, and I feel much more than grateful to Professor Yuan for his valuable guidance and suggestions. I thank Professor M. J. D. Powell for the encouragement and helpful discussions. Professor Powell helped improve the proof of Theorem 3.1. This work was partly done during a visit to Professor Klaus Schittkowski at Universität Bayreuth in 2010. I thank Alexander von Humboldt Foundation for supporting this visit, and I am much grateful to Professor Schittkowski for his warm hospitality. I appreciate the help of Professor Andrew R. Conn and an anonymous referee, whose comments have substantially improved the paper.

Author information

Authors and Affiliations

  1. Institute of Computational Mathematics and Scientific/Engineering Computing, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, P.O. Box 2719, Beijing, 100190, China

    Zaikun Zhang

Authors
  1. Zaikun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zaikun Zhang.

Additional information

Partially supported by Chinese NSF grants 10831006, 11021101, CAS grant kjcx-yw-s7, and Alexander von Humboldt Foundation.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, Z. Sobolev seminorm of quadratic functions with applications to derivative-free optimization. Math. Program. 146, 77–96 (2014). https://doi.org/10.1007/s10107-013-0679-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10107-013-0679-3

Keywords

Mathematics Subject Classification (2010)

Search

Navigation