Abstract
Artificial feedforward neural networks have received researchers’ great interest due to its ability to approximate functions without having a prior knowledge about the true underlying function. The most popular algorithm for training these networks is the backpropagation algorithm that is based on the minimization of the mean square error cost function. However this algorithm is not robust in the presence of outliers that may pollute the training data. In this chapter we present several methods to robustify neural network training algorithms. First, employing a family of robust statistics estimators, commonly known as M-estimators, in the backpropagation algorithm is reviewed and evaluated for the task of function approximation and dynamical model identification. As theseM-estimators sometimes do not have sufficient insensitivity to data outliers, the chapter next resorts to the statistically more robust estimator of the least median of squares, and develops a stochastic algorithm to minimize a related cost function. The reported experimental results have indeed shown the improved robustness of the new algorithm, especially compared to the standard backpropagation algorithm, on datasets with varying degrees of outlying data.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Pernia-Espinoza, A.V., Ordieres-Mere, J.B., Martinez-de-Pison, F.J., Gonzalez-Marcos, A.: TAO-robust backpropagation learning algorithm. Neural Networks 18, 1–14 (2005)
Annema, A.-J.: Feed-forward neural networks: Vector decomposition analysis, Modelling and Analog Implementation. Kluwer Academic Publishers, Boston (1995)
Baxt, W.G.: Use of an artificial neural network for data analysis in clinical decision making: The diagnosis of acute coronary occlusion. Neural Computation 2, 480–489 (1990)
Baxt, W.G.: Use of an artificial neural network for the diagnosis of myocardial infarction. Annals of Internal Medicine 115, 843–848 (1991)
Bishop, C.M.: Neural networks for pattern recognition. Clarendon Press, Oxford (1995)
Goodall, C.: M-estimators of location: An outline of the theory. In: Hoaglin, Mosteller, Turkey (eds.) Understanding Robust and Exploratory Data Analysis, pp. 339–403 (1983)
Peterson, C., Andrson, J.: A mean field theory learning algorithm for neural networks. Complex Systems 1(1), 995–1019 (1987)
Chuang, C.C., Su, S.F., Hsiao, C.C.: The annealing robust backpropagation (ARBP) learning algorithm. IEEE Trans. on Neural Networks 11(5), 1067–1077 (2000)
Churchland, P.S., Sejnowski, T.J.: The computational brain in deutscher sprache. In: Comutational intelligence. Vieweg Verlag (1997)
Corana, A., Marchesi, M., Martini, C., Ridella, S.: Minimizing multimodal functions of continuous variables with the simulated annealing algorithm. ACM Trans. on Mathematical Software 13(3), 262–280 (1987)
Cowan, J.D.: Neural networks: The early days. In: Touretzky, D. (ed.) Advances in neural information processing systems 2 (NIPS), pp. 828–842. Morgan Kaufmann, San Francisco (1990)
Rumelhart, D.E., McClelland, J.L.: Parallel distributed processing. MIT Press, Cambridge (1986)
Van den Bout, D.E., Miller, T.K.: Graph partitioning using annealed networks. IEEE Trans. on Neural Networks 1, 192–203 (1990)
Aarts, E.H., Korst, J.: Simulated annealing and Boltzmann machines: stochastic approach to combinatorial optimization and neural computing. John Wiley and Sons, Inc., New York (1989)
Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics, The approach based on influence functions. Wiley, NewYork (1986)
Dreyfus, G.: Neural networks methodology and applications. Springer, Heidelberg (2005)
Bibro, G.L., Synder, W.E., Garnier, S.J., Gault, J.W.: Mean field annealing: A formalism for constructing GNC-like algorithms. IEEE Trans. on Neural Networks 3, 131–138 (1992)
Goffe, W.L., Ferrier, G.D., Rogers, J.: Global optimization of statistical functions with simulated annealing. Journal of Econometrics 60, 65–99 (1994)
Gupta, P., Sinha, N.: An improved approach for nonlinear system identification using neural networks. Journal of the Franklin Institute 336(4), 721–734 (1999)
Haykin: Neural Networks: A comprehensive foundation, 2nd edn. Macmillan College Publishing, New York (1994)
Hertz, J., Krogh, A., Palmer, R.G.: Introduction to the theory of Neural Computation. Addison Vesley, New York (1991)
Hornik, K.: Multi-layer Feed-Forward Networks are Universal Approximators. In: White, H., et al. (eds.) Artificial Neural Networks: approximation and Learning Theory. Blackwell publishers, Cambridge (1992)
Hutchinson, J.M.: A radial basis function approach to financial time series analysis. Ph.D. dissertation, Massachusetts Institute of Technology (1994)
Moody, J., Darken, C.: Fast learning in networks of locally-tuned processing units. Neural Computa 1, 281–284 (1989)
Liano, K.: Robust error measure for supervised neural network learning with outliers. IEEE Trans. Neural Networks 7, 246–250 (1996)
Kenneth, D., Kahng, A.B.: Simulated annealing of neural networks: The cooling strategy reconsidered. Technical report CA90024, UCLA Computer science Dept., Los Angeles (1965)
Kumpati, S., Narendra: Identification and control of dynamical systems using neural networks. IEEE Trans. on Neural Networks 1(1), 4–27 (1990)
Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Networks 4(2), 251–257 (1991)
Huang, L., Zhang, B.L., Huang, Q.: Robust interval regression analysis using neural network. Fuzzy Sets Syst., 337–347 (1998)
Le Cun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems, vol. 2, pp. 248–257 (1990)
Leung, M.T., Engeler, W.E., Frank, P.: Fingerprint processing using backpropagation neural networks. In: Proceedings of the International Joint Conference on Neural Networks I, pp. 15–20 (1990)
Masters, T.: Advanced algorithms for neural networks: A C++ source book. John Wiley and Sons, Inc., New York (1995)
Hassoun, M.H.: Fundamentals of artificial neural networks. MIT Press, Cambridge (1995)
Huber, P.J.: Robust Statistics. John Wiley and Sons, New York (1981)
Rousseeuw, P.J., Leroy, A.M.: Robust regression and outlier detection. Wiley, New York (1987)
Pomerleau, D.A.: Neural network perception for mobile robot guidance. Kluwer, Boston (1993)
Rosenblatt, F.: The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 65, 386–408 (1959)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by propagating errors. Nature 323, 533–546 (1986)
Welch, R.M., Sengupta, S.K., Goroch, A.K., Rabindra, P., Rangaraj, N., Navar, M.S.: Polar cloud and surface classification using AVHRR imagery: An intercomparison of methods. Journal of Applied meteorology 31, 405–420 (1992)
William, J.J.: Introduction to robust and quasi-robust statistical methods. Springer, Heidelberg (1983)
Zamarreno, J.M., Vega, P.: State space neural network: Properties and application. Neural Networks 11(6), 1099–1112 (1998)
Zaprains, A.D., Refenes, A.P.: Principles of neural model identification, selection, and adequacy with applications to financial econometrics. In: Perspective in Neuro Computing. Springer, London (1999)
Zhang, Z.: Parameter estimation techniques: A tutorial with application to conic fitting. Image and Vision Computing Journal 15(1), 59–76 (1997)
Rusiecki, A.L.: Robust learning algorithm with the variable learning rate. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds.) ICAISC 2006. LNCS, vol. 4029, pp. 83–90. Springer, Heidelberg (2006)
Rusiecki, A.L.: Robust LTS backpropagation learning algorithm. In: Sandoval, F., Prieto, A.G., Cabestany, J., Graña, M. (eds.) IWANN 2007. LNCS, vol. 4507, pp. 102–109. Springer, Heidelberg (2007)
Chuang, C., Jeng, J., Lin, P.: Annealing robust radial basis function networks for function approximation with outliers. Neurocomputing 56, 123–139 (2004)
Lee, C., Chung, P., Tsai, J., Chang, C.: Robust radial basis function neural networks. IEEE Trans. Systems, Man, and Cybernetics – Part B: Cybernetics 29(6), 674–685 (1999)
Rusiecki, A.L.: Robust MCD-based backpropagation learning algorithm. In: Rutkowski, et al. (eds.) ICAISC 2008. LNCS, vol. 5097, pp. 154–163. Springer, Heidelberg (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
El-Melegy, M.T., Essai, M.H., Ali, A.A. (2009). Robust Training of Artificial Feedforward Neural Networks. In: Hassanien, AE., Abraham, A., Vasilakos, A.V., Pedrycz, W. (eds) Foundations of Computational, Intelligence Volume 1. Studies in Computational Intelligence, vol 201. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-01082-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-01082-8_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-01081-1
Online ISBN: 978-3-642-01082-8
eBook Packages: EngineeringEngineering (R0)