Advertisement

Projective Approximation Based Quasi-Newton Methods

  • Alexander SenovEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10710)

Abstract

We consider a problem of optimizing convex function of vector parameter. Many quasi-Newton optimization methods require to construct and store an approximation of Hessian matrix or its inverse to take function curvature into account, thus imposing high computational and memory requirements. We propose four quasi-Newton methods based on consecutive projective approximation. The idea of these methods is to approximate the product of the function Hessian inverse and function gradient in a low-dimensional space using appropriate projection and then reconstruct it back to original space as a new direction for the next estimate search. By exploiting Hessian rank deficiency in a special way it does not require to store Hessian matrix neither its inverse thus reducing memory requirements. We give a theoretical motivation for the proposed algorithms and prove several properties of corresponding estimates. Finally, we provide a comparison of the proposed methods with several existing ones on modelled data. Despite the fact that the proposed algorithms turned out to be inferior to the limited memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) one, they have important advantage of being easy to extent and improve. Moreover, two of them do not require the function gradient knowledge.

Keywords

Least-squares Function approximation Convex optimization Iterative methods Quadratic programming Quasi-Newton methods Projective methods 

Notes

Acknowledgments

This work was supported by Russian Science Foundation (project 16-19-00057).

References

  1. 1.
    Box, G.E., Draper, N.R., et al.: Empirical Model-Building and Response Surfaces. Wiley, New York (1987)zbMATHGoogle Scholar
  2. 2.
    Boyd, S.: Global optimization in control system analysis and design. In: Control and Dynamic Systems V53: High Performance Systems Techniques and Applications: Advances in Theory and Applications, vol. 53, p. 1 (2012)Google Scholar
  3. 3.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefGoogle Scholar
  4. 4.
    Broyden, C.G.: The convergence of a class of double-rank minimization algorithms 1. General considerations. IMA J. Appl. Math. 6(1), 76–90 (1970)CrossRefGoogle Scholar
  5. 5.
    Conn, A.R., Gould, N.I., Toint, P.L.: Convergence of quasi-Newton matrices generated by the symmetric rank one update. Math. Program. 50(1–3), 177–195 (1991)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Davidon, W.C.: Variable metric method for minimization. SIAM J. Optim. 1(1), 1–17 (1991)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theor. 52(4), 1289–1306 (2006)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, New York (1987)zbMATHGoogle Scholar
  9. 9.
    Ford, J., Moghrabi, I.: Multi-step quasi-Newton methods for optimization. J. Comput. Appl. Math. 50(1–3), 305–323 (1994)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Forrester, A., Keane, A.: Recent advances in surrogate-based optimization. Prog. Aerosp. Sci. 45(1), 50–79 (2009)CrossRefGoogle Scholar
  11. 11.
    Granichin, O., Volkovich, Z.V., Toledano-Kitai, D.: Randomized Algorithms in Automatic Control and Data Mining, vol. 67. Springer, Heidelberg (2015).  https://doi.org/10.1007/978-3-642-54786-7CrossRefGoogle Scholar
  12. 12.
    Granichin, O.N.: Stochastic approximation search algorithms with randomization at the input. Autom. Remote Control 76(5), 762–775 (2015)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Hoffmann, W.: Iterative algorithms for Gram-Schmidt orthogonalization. Computing 41(4), 335–348 (1989)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Krause, A.: SFO: a toolbox for submodular function optimization. J. Mach. Learn. Res. 11, 1141–1144 (2010)zbMATHGoogle Scholar
  15. 15.
    Nesterov, Y.: Introductory Lectures on Convex Programming Volume I: Basic Course. Citeseer (1998)Google Scholar
  16. 16.
    Nocedal, J.: Updating quasi-Newton matrices with limited storage. Math. Comput. 35(151), 773–782 (1980)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Polyak, B.T.: Introduction to optimization. Translations series in mathematics and engineering. Optimization Software Inc., Publications Division, New York (1987)Google Scholar
  18. 18.
    Senov, A.: Accelerating gradient descent with projective response surface methodology. In: Battiti, R., Kvasov, D.E., Sergeyev, Y.D. (eds.) LION 2017. LNCS, vol. 10556, pp. 376–382. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-69404-7_34CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Control of Complex Systems LaboratoryInstitute of Problems of Mechanical EngineeringSt. PetersburgRussia

Personalised recommendations