Abstract
A simple modification technique is introduced to the limited memory BFGS (L-BFGS) method for solving large-scale nonlinear least-squares problems. The L-BFGS method computes a Hessian approximation of the objective function implicitly as the outcome of updating a basic matrix, \(H_k^0\) say, in terms of a number of pair vectors which are available from most recent iterations. Using the features of the nonlinear least-squares problem, we consider certain modifications of the pair vectors and propose some alternative choices for \(H_k^0\), instead of the usual multiple of the identity matrix. We also consider the possibility of using part of the Gauss-Newton Hessian which is available on each iteration but cannot be stored explicitly. Numerical results are described to show that the proposed modified L-BFGS methods perform substantially better than the standard L-BFGS method.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al-Baali, M.: New initial Hessian approximations for the limited memory BFGS method for large scale optimization. J. Fac. Sci. UAE Univ 14, 167–175 (1995)
Al-Baali, M.: Quasi-Newton algorithms for large-scale nonlinear least-squares. In: High Performance Algorithms and Software for Nonlinear Optimization, pp. 1–21. Springer, Boston (2003)
Al-Baali, M.: Damped techniques for enforcing convergence of quasi-Newton methods. Optim. Methods Softw. 29(5), 919–936 (2014)
Al-Baali, M., Conforti, D., Musmanno, R.: Computational experiments with scaled initial Hessian approximation for the Broyden family methods. Optimization 48(3), 375–389 (2000)
Al-Baali, M., Fletcher, R.: Variational methods for non-linear least-squares. J. Oper. Res. Soc. 36(5), 405–421 (1985)
Bouaricha, A., Moré, J.J.: Impact of partial separability on large-scale optimization. Comput. Optim. Appl. 7(1), 27–40 (1997)
Byrd, R. H., Nocedal, J., Zhu, C.: Towards a discrete Newton method with memory for large-scale optimization. In: Nonlinear Optimization and Applications, pp. 1–12. Springer, Boston (1996)
Dener, A., Munson, T.: Accelerating limited-memory quasi-Newton convergence for large-scale optimization. In: International Conference on Computational Science, pp. 495–507. Springer, Cham (2019)
Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)
Gilbert, J.C., Lemaréchal, C.: Some numerical experiments with variable-storage quasi-Newton algorithms. Math. Program. 45(1–3), 407–435 (1989)
Huschens, J.: On the use of product structure in secant methods for nonlinear least squares problems. SIAM J. Optim. 4(1), 108–129 (1994)
Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45(1–3), 503–528 (1989)
Nocedal, J.: Updating quasi-Newton matrices with limited storage. Math. Comput. 35(151), 773–782 (1980)
Nocedal, J., Wright, S.: Numerical Optimization. Springer, New York (2006)
Oren, S.S., Spedicato, E.: Optimal conditioning of self-scaling variable metric algorithms. Math. Program. 10(1), 70–90 (1976)
Yabe, H., Takahashi, T.: Numerical comparison among structured quasi-Newton methods for nonlinear least squares problems. J. Oper. Res. Soc. Jpn. 34(3), 287–305 (1991)
Zou, X., Navon, I.M., Berger, M., Phua, K.H., Schlick, T., Le Dimet, F.X.: Numerical experience with limited-memory quasi-Newton and truncated Newton methods. SIAM J. Optim. 3(3), 582–608 (1993)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Al-Siyabi, A., Al-Baali, M. (2020). New Basic Hessian Approximations for Large-Scale Nonlinear Least-Squares Optimization. In: Vasant, P., Zelinka, I., Weber, GW. (eds) Intelligent Computing and Optimization. ICO 2019. Advances in Intelligent Systems and Computing, vol 1072. Springer, Cham. https://doi.org/10.1007/978-3-030-33585-4_59
Download citation
DOI: https://doi.org/10.1007/978-3-030-33585-4_59
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33584-7
Online ISBN: 978-3-030-33585-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)