Abstract
Some highly successful algorithms for unconstrained minimization without derivatives construct changes to the variables by applying trust region methods to quadratic approximations to the objective function \({F (\underline{x}), \underline{x} \in \mathcal{R}^n}\). A quadratic model has (n + 1) (n + 2)/2 independent parameters, but each new model may interpolate only 2n + 1 values of F, for instance. The symmetric Broyden method takes up the remaining freedom by minimizing the Frobenius norm of the difference between the second derivative matrices of the old and new models, which usually works well in practice. We consider an extension of this technique that combines changes in first derivatives with changes in second derivatives. A simple example suggests that the extension does bring some advantages, but numerical experiments on three test problems with up to 320 variables are disappointing. On the other hand, rates of convergence are investigated numerically when F is a homogeneous quadratic function, which allows very high accuracy to be achieved in practice, the initial and final errors in the variables being about 10 and 10−5000, respectively. It is clear in some of these experiments that the extension does reduce the number of iterations. The main difficulty in the work was finding a way of implementing the extension sufficiently accurately in only \({\mathcal{O}( n^2 )}\) operations on each iteration. A version of the truncated conjugate gradient procedure is suitable, that is used in the numerical experiments, and that is described in detail in an appendix.
Similar content being viewed by others
References
Broyden C.G., Dennis J.E., Moré J.J.: On the local and superlinear convergence of quasi-Newton methods. J. Inst. Math. Appl. 12, 223–245 (1973)
Buckley, A.G.: Test functions for unconstrained minimization. Technical Report 1989 CS-3, Dalhousie University, Canada (1989)
Conn A.R., Gould N.I.M., Lescrenier M., Toint Ph.L.: Performance of a multifrontal scheme for partially separable optimization. In: Gomez, S., Hennart, J.-P. (eds) Advances in Optimization and Numerical Analysis, pp. 79–96. Kluwer, Dordrecht (1994)
Conn A.R., Scheinberg K., Vicente L.N.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)
Fletcher R.: Practical Methods of Optimization. Wiley, Chichester (1987)
Powell M.J.D.: Least Frobenius norm updating of quadratic models that satisfy interpolation conditions. Math. Program. B 100, 183–215 (2004)
Powell M.J.D.: On updating the inverse of a KKT matrix. In: Yuan, Y. (eds) Numerical Linear Algebra and Optimization, pp. 56–78. Science Press, Beijing (2004)
Powell M.J.D.: The NEWUOA software for unconstrained optimization without derivatives. In: Di Pillo, G., Roma, M. (eds) Large-Scale Optimization, pp. 255–297. Springer, New York (2006)
Powell, M.J.D.: The BOBYQA algorithm for bound constrained optimization without derivatives. Report No. DAMTP 2009/NA06, CMS, University of Cambridge (2009)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Powell, M.J.D. Beyond symmetric Broyden for updating quadratic models in minimization without derivatives. Math. Program. 138, 475–500 (2013). https://doi.org/10.1007/s10107-011-0510-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-011-0510-y