Abstract
This chapter deals with fundamental nonlinear optimization techniques that strive to find a local minimum of a possible multimodal loss function. Most of this chapter focuses on unconstrained optimization, but some basics also deal with constrained optimization. First, the exact criteria to optimize are investigated: Batch adaptation, sample adaptation, and mini-batch adaptation as a way in between are discussed. The role of the initial parameters as the starting point for a local search is explained. Existing methods for local nonlinear optimization can be separated into two classes: (i) direct search approach and (ii) gradient-based algorithms. The latter can again be subdivided into general and nonlinear least squares methods. The ideas of the most important algorithms are explained – no algorithmic details are given. The reader/user will understand which approach possesses which properties and thus might be suited for a specific problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barnard, E.: Optimization for training neural nets. IEEE Trans. Neural Netw. 3(2), 322–325 (1992)
Battiti, R.: First and second order methods for learning: between stepest descent and Newton’s method. Neural Comput. 4(2), 141–166 (1992)
Bishop, C.M.: Neural Networks for Pattern Recognition. Clarendon Press, Oxford (1995)
Branch, M.A., Grace, A.: MATLAB Optimization Toolbox User’s Guide, Version 1.5. The MATHWORKS Inc., Natick, MA (1998)
Charalambous, C.: Conjugate gradient algorithm for efficient training of artificial neural networks. IEE Proc.-G 139(3), 301–310 (1992)
Golub, G.H., Van Loan, C.F.: Matrix Computations. Mathematical Sciences. The Johns Hopkins University Press, Baltimore (1987)
Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Netw. 5(6), 989–993 (1994)
Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79 (1982)
Reklaitis, G.V., Ravindran, A., Ragsdell, K.M.: Engineering Optimization – Methods and Applications. John Wiley & Sons, London (1983)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Rumelhart, D.E., McClelland, J.L. (eds.) Parallel Distributed Processing: Explorations in the Mircostructure of Cognition, vol. 1, chapter 8. MIT Press, Cambridge (1986)
Scales, L.E.: Introduction to Non-Linear Optimization. Computer and Science Series. Macmillan, London (1985)
Vanderplaats, G.N.: Numerical Optimization Techniques for Engineering Design. Series in Mechanical Engineering. McGraw-Hill, New York (1984)
Werbos, P.J.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioural Sciences. Ph.D. thesis, Harvard University, Boston, USA (1974)
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Nelles, O. (2020). Nonlinear Local Optimization. In: Nonlinear System Identification. Springer, Cham. https://doi.org/10.1007/978-3-030-47439-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-47439-3_4
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-47438-6
Online ISBN: 978-3-030-47439-3
eBook Packages: Physics and AstronomyPhysics and Astronomy (R0)