Abstract
Supervised learning in neural networks based on the popular backpropagation method can be often trapped in a local minimum of the error function. The class of backpropagation-type training algorithms includes local minimization methods that have no mechanism that allows them to escape the influence of a local minimum. The existence of local minima is due to the fact that the error function is the superposition of nonlinear activation functions that may have minima at different points, which sometimes results in a nonconvex error function. This work investigates the use of global search methods for batch-mode training of feedforward multilayer perceptrons. Global search methods are expected to lead to “optimal” or “near-optimal” weight configurations by allowing the neural network to escape local minima during training and, in that sense, they improve the efficiency of the learning process. The paper reviews the fundamentals of simulated annealing, genetic and evolutionary algorithms as well as some recently proposed deflection procedures. Simulations and comparisons are presented.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
E.K. Blum, “Approximation of Boolean functions by sigmoidal networks: Part I: XOR and other two variable functions”, Neural Computation, vol. 1, 1989, 532–540.
M. Burton Jr. and G.J. Mpitsos, “Event dependent control of noise enhances learning in neural networks”, Neural Networks, vol. 5, 1992, 627–637.
A. Corana, M. Marchesi, C. Martini, and S. Ridella, “Minimizing multimodal functions of continuous variables with the Simulated Annealing algorithm”, ACM Trans. Math. Soft., vol. 13, 1987, 26 2280.
M. Gori and A. Tesi, “On the problem of local minima in backpropagation”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, 1992, 76–85.
J.H. Holland, “Adaptation in Neural and Artificial Systems”, University of Michigan Press, 1975.
C. Houck, J. Joines, and M. Kay, “A Genetic Algorithm for Function Optimization: A Matlab Implementation”, NCSU-IE TR, 1995, 9509.
S. Kirkpatrick, C.D. Gelatt Jr., and M.P. Vecchi, “Optimization by simulated annealing”, Science, vol. 220, 1983, 671–680.
Y. Lee, S.H. Oh, and M. Kim, “An analysis of premature saturation in backpropagation learning”, Neural Networks, vol.6, 9993, 719–728.
G.D. Magoulas, M.N. Vrahatis, and G.S. Androulakis, “Xffective backpropagation training with variable stepsize”, Neural Networks, vol.10, No.1, 6997, 63–12.
G.D. Magoulas, M.N. Vrahatis, and G.S. Androulakis, “On the alleviation of local minimi in backpropagation”, Nonlinear Analysis, Theory, Methods and Applications, vol. 30, 1997, 4545–4550.
G.D. Magoulas, M.N. Vrahatis, T.N. Grapsa, and G.S. Androulakis, “Neural network supervised training based on a dimension reducing method”, Mathematics of Neural Networks, Models, Algorithms and Applications, S.W. Ellacott, J.C. Mason, L.J. Anderson Eds., Kluwer Academic Publishers, Boston, 1997, 245–249.
Z. Michalewicz, “Genetic algorithms + data structures = evolution programs”, Springer, 1996.
M.F. Möller, “A scaled conjugate gradient algorithm for fast supervised learning”, Neural Networks, vol. 6, 1993, 525–533.
V.P. Plagianakos and M.N. Vrahatis, “Training neural networks with 3-bit integer weights”, Proceedings of Genetic and Evolutionary Computation Conference (GECCO’99), 1999, 910–915.
V.P. Plagianakos and M.N. Vrahatis, “Neural network training with constrained integer weights”, Proceedings of Congress on Evolutionary Computation (CEC’99), 1999, 2007–2013.
V.P. Plagianakos and M.N. Vrahatis, “Training Neural Networks with Threshold Activation Functions and Constrained Integer Weights”, Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN’2000), (2000).
D.E. Rumelhart. G.E. Hinton, and R.J. Williams, “Learning internal representations by error propagation”, Parallel Distributed Processing: Explorations in the Microstructure of Cognition 1, D.E. Rumelhart, J.L. McClelland Eds., MIT Press, 1986, 318–362.
R. Storn and K. Price, “Differential Evolution-A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces”, Journal of Global Optimization, vol. 11, 1997, 341–359.
P.P. Van der Smagt, “Minimisation methods for training feedforward neural networks”, Neural Networks, vol. 7, 1994, 1–11.
T.P. Vogl, J.K. Mangis, A.K. Rigler, W.T. Zink, and D.L. Alkon, “Accelerating the convergence of the back-propagation method”, Biological Cybernetics, vol. 59, 1988, 257–263.
S.T. Weslstead, “Neural network and fuzzy logic applications in C/C++”, Wiley, 1994.
X.-H. Yu, G.-A. Chen, “On the local minima free condition of back-propagation learning”, IEEE Trans. Neural Networks, vol. 6, 1995, 1300–1303.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Kluwer Academic Publishers
About this chapter
Cite this chapter
Plagianakos, V.P., Magoulas, G.D., Vrahatis, M.N. (2001). Supervised Training Using Global Search Methods. In: Hadjisavvas, N., Pardalos, P.M. (eds) Advances in Convex Analysis and Global Optimization. Nonconvex Optimization and Its Applications, vol 54. Springer, Boston, MA. https://doi.org/10.1007/978-1-4613-0279-7_26
Download citation
DOI: https://doi.org/10.1007/978-1-4613-0279-7_26
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-7923-6942-4
Online ISBN: 978-1-4613-0279-7
eBook Packages: Springer Book Archive