Soft Computing

, Volume 18, Issue 8, pp 1603–1614 | Cite as

Hybrid back-propagation training with evolutionary strategies

Methodologies and Application

Abstract

This work presents a hybrid algorithm for neural network training that combines the back-propagation (BP) method with an evolutionary algorithm. In the proposed approach, BP updates the network connection weights, and a (\(1+1\)) Evolutionary Strategy (ES) adaptively modifies the main learning parameters. The algorithm can incorporate different BP variants, such as gradient descent with adaptive learning rate (GDA), in which case the learning rate is dynamically adjusted by the stochastic (\(1+1\))-ES as well as the deterministic adaptive rules of GDA; a combined optimization strategy known as memetic search. The proposal is tested on three different domains, time series prediction, classification and biometric recognition, using several problem instances. Experimental results show that the hybrid algorithm can substantially improve upon the standard BP methods. In conclusion, the proposed approach provides a simple extension to basic BP training that improves performance and lessens the need for parameter tuning in real-world problems.

Keywords

Neural networks Back-propagation  Evolutionary strategies 

References

  1. Alba E, Chicano JF (2004) Training neural networks with GA hybrid algorithms. In: Deb K, et al (eds), GECCO (1). Lecture Notes in computer science, vol 3102. Springer, Berlin, pp 852–863Google Scholar
  2. Ayinde O, Yang Y (2002) Face recognition approach based on rank correlation of Gabor-filtered images. Pattern Recognit 36(6):1275–1289CrossRefGoogle Scholar
  3. Cantú-Paz E, Kamath C (2005) An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Syst Man Cybern Soc 35(5):915–927CrossRefGoogle Scholar
  4. Castillo O, Melin P (2002) Hybrid intelligent systems for time series prediction using neural networks, fuzzy logic, and fractal theory. IEEE Trans Neural Netw 13(6):1395–1408CrossRefGoogle Scholar
  5. Cortez P, Cerdeira A, Almeida F, Matos T, Reis J (2009) Modeling wine preferences by data mining from physicochemical properties. Decis Support Syst 47(4):547–553CrossRefGoogle Scholar
  6. DeJong K (2002) Evolutionary computation: a unified approach. The MIT Press, CambridgeGoogle Scholar
  7. Denoeux T (2000) A neural network classifier based on Dempster–Shafer theory. IEEE Syst Man Cybern Soc 30(2):131–150CrossRefMathSciNetGoogle Scholar
  8. Dogan N, Tanrikulu Z (2010) A comparative framework for evaluating classification algorithms. In: Proceedings of the world congress on engineering 2010, vol 1. WCE, pp 379–384Google Scholar
  9. Dorigo M, Stützle T (2004) Ant colony optimization. Bradford Company, ScituateGoogle Scholar
  10. Eiben AE, Smith JE (2003) Introduction to evolutionary computing. Springer, BerlinGoogle Scholar
  11. Fogel DB, Fogel LJ, Porto VW (1990) Evolving neural networks. Biol Cybern 63(6):487–493CrossRefGoogle Scholar
  12. Hagan M, Demuth H, Beale M (1996) Neural Network Design. PWS Publishing Company, BostonGoogle Scholar
  13. Harp SA, Samad T, Guha A (1989) Towards the genetic synthesis of neural network. In: Proceedings of the third international conference on Genetic algorithms. Morgan Kaufmann Publishers Inc., San Francisco, pp 360–369Google Scholar
  14. Hoel PG, Port SC, Stone CJ (1987) Introduction to stochastic processes. Waveland Press, Long GroveGoogle Scholar
  15. Husbands P (1994) Distributed coevolutionary genetic algorithms for multi-criteria and multi-constraint optimisation. In: Selected papers from AISB workshop on evolutionary computing. Springer, London, pp 150–165Google Scholar
  16. Isasi Viñuela P (2004) Redes neuronales artificiales: un enfoque práctico. TPearson Educacion, Upper Saddle RiverGoogle Scholar
  17. Kim HB, Jung SH, Kim TG, Park KH (1996) Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates. Neurocomputing 11(1):101–106CrossRefMATHGoogle Scholar
  18. Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2009) Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw 22(10):1448–1462CrossRefGoogle Scholar
  19. Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680CrossRefMATHMathSciNetGoogle Scholar
  20. Lee S-W (1996) Off-line recognition of totally unconstrained handwritten numerals using multilayer cluster neural network. IEEE Trans Pattern Anal Mach Intell 18(6):648–652CrossRefGoogle Scholar
  21. Mackey M, Glass L (1977) Oscillation and chaos in physiological control systems. Science 197(4300):287–289CrossRefGoogle Scholar
  22. Melin P, Castillo O (2005) Hybrid intelligent systems for pattern recognition using soft computing: an evolutionary approach for neural networks and fuzzy systems (Studies in fuzziness and soft computing). Springer-Verlag New York Inc., SecaucusGoogle Scholar
  23. Melin P, Soto J, Castillo O, Soria J (2012) A new approach for time series prediction using ensembles of ANFIS models. Expert Syst Appl 39(3):3494–3506CrossRefGoogle Scholar
  24. Merelo J, Patón M, Cañas A, Prieto A, Morán F (1993) Optimization of a competitive learning neural network by genetic algorithms. In: Proceedings of the international workshop artificial neural networks (IWANN93). Lecture notes in computer science, vol 686. Morgan Kaufmann Publishers Inc., Berlin, pp 185–192Google Scholar
  25. Miller GF, Todd PM, Hegde SU (1989) Designing neural networks using genetic algorithms. In: Proceedings of the third international conference on Genetic algorithms. Morgan Kaufmann Publishers Inc., San Francisco, pp 379–384Google Scholar
  26. Nolfi S, Floreano D (2000) Evolutionary robotics: the biology, intelligence, and technology. MIT Press, CambridgeGoogle Scholar
  27. Patel D (1996) Using genetic algorithms to construct a network for financial prediction. In: Proceedings of SPIE: applications of artificial neural networks in image processing. pp 204–213Google Scholar
  28. Radi A, Poli R (2003) Discovering efficient learning rules for feedforward neural networks using genetic programming, chap 7. Physica-Verlag GmbH, Heidelberg, pp 133–159Google Scholar
  29. Refaeilzadeh P, Tang L, Liu H (2009) Cross-validation. In: Liu L, Özsu MT (eds) Encyclopedia of database systems. Springer, Berlin, pp 532–538Google Scholar
  30. Riedmiller M (1994) Rprop—description and implementation details. Technical report, University of KarlsruheGoogle Scholar
  31. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation, chap 8. MIT Press, Cambridge, pp 318–362Google Scholar
  32. Samanta B (2011) Prediction of chaotic time series using computational intelligence. Expert Syst Appl 38(9):11406–11411CrossRefMathSciNetGoogle Scholar
  33. Samarasinghe S (2006) Neural networks for applied sciences and engineering. Auerbach Publications, BostonGoogle Scholar
  34. Samaria FS, Harter AC (2002) Parameterisation of a stochastic model for human face identification. In: Applications of computer vision, vol 1994. Proceedings of the second IEEE workshop on Sarasota, FL. pp 138–142Google Scholar
  35. Schwefel H-P (1981) Numerical optimization of computer models. John Wiley & Sons Inc., New YorkMATHGoogle Scholar
  36. Stanley KO, D’Ambrosio DB, Gauci J (2009) A hypercube-based encoding for evolving large-scale neural networks. Artif Life 15(2):185–212CrossRefGoogle Scholar
  37. Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127CrossRefGoogle Scholar
  38. Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. Trans Evol Comp 1(1):67–82CrossRefGoogle Scholar
  39. Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447CrossRefGoogle Scholar
  40. Zhang G (2000) Neural networks for classification: a survey. IEEE Syst Man Cybern Soc 30(4):451–462 Google Scholar
  41. Zhong P, Fukushima M (2007) Regularized nonsmooth newton method for multi-class support vector machines. Optim Methods Softw 22(1):225–236CrossRefMATHMathSciNetGoogle Scholar
  42. Zimmerman DW, Williams RH (1986) Modern elementary statistics, with theoretical supplement and BASIC programming. American Sciences Press, SyracuseMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • José Parra
    • 1
  • Leonardo Trujillo
    • 1
  • Patricia Melin
    • 1
  1. 1.Instituto Tecnológico de TijuanaTijuanaMéxico

Personalised recommendations