Soft Computing

, Volume 19, Issue 3, pp 679–689 | Cite as

Training neural networks via simplified hybrid algorithm mixing Nelder–Mead and particle swarm optimization methods

  • Shih-Hui Liao
  • Jer-Guang Hsieh
  • Jyh-Yeong Chang
  • Chin-Teng Lin
Methodologies and Application
First Online:

Abstract

In this paper, a new and simplified hybrid algorithm mixing the simplex method of Nelder and Mead (NM) and particle swarm optimization algorithm (PSO), abbreviated as SNM-PSO, is proposed for the training of the parameters of the Artificial Neural Network (ANN). Our method differs from other hybrid PSO methods in that, \(n+1\) particles, where \(n\) is the dimension of the search space, are randomly selected (without sorting), at each iteration of the proposed algorithm for use as the initial vertices of the NM algorithm, and each such particle is replaced by the corresponding final vertex after executing the NM algorithm. All the particles are then updated using the standard PSO algorithm. Our proposed method is simpler than other similar hybrid PSO methods and places more emphasis on the exploration of the search space. Some simulation problems will be provided to compare the performances of the proposed method with PSO and other similar hybrid PSO methods in training an ANN. These simulations show that the proposed method outperforms the other compared methods.

Keywords

Artificial Neural Network (ANN) Particle swarm optimization (PSO) Simplex method of Nelder and Mead (NM) 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

This work was supported in part by the Aiming for the Top University Plan of National Chiao Tung University, the Ministry of Education, Taiwan, under Grant Number 101W9633 & 103W963, in part by the UST-UCSD International Center of Excellence in Advanced Bio-engineering sponsored by the Taiwan National Science Council I-RiCE Program under Grant Number: NSC 102-2911-I-009-101.

References

  1. Ahmadi MA, Ebadi M, Shokrollahi A, Majid SMJ (2013) Evolving artificial neural network and imperialist competitive algorithm for prediction oil flow rate of the reservoir. Appl Soft Comput 13(3):1085–1098CrossRefGoogle Scholar
  2. Blake C, Merz C (1998) UCI Repository of Machine Learning Databases. http://www.ics.uci.edu/~mlearn/MLRepository.html
  3. Bertsekas D (1999) Nonlinear Programming. Athena Scientific, BelmontMATHGoogle Scholar
  4. Chen J, Tu X, Fu J (2005) Multilayered feed forward neural network based on particle swarm optimizer algorithm. J Syst Eng Electron 16(4):682–686Google Scholar
  5. Chang CC, Lin CJ (2001) LIBSVM: A Library for Support Vector Machines. http://www.csie.ntu.edu.tw/~cjlin/libsvm
  6. Chau KW (2007) Application of a PSO-based neural network in analysis of outcomes of construction claims. Autom Constr 16(5):642–646Google Scholar
  7. Civicioglu P, Besdok E (2011) A conceptual comparison of the Cuckoo-search, particle swarm optimization, differential evolution and artificial bee colony algorithms. Artif Intell Rev 39(5):315–346Google Scholar
  8. Da Y, Ge XR (2005) An improved PSO-based ANN with simulated annealing technique. Neurocomputing 63:527–533CrossRefGoogle Scholar
  9. Deng W, Li W, Yang XH (2011) A novel hybrid optimization algorithm of computational intelligence techniques for highway passenger volume prediction. Expert Syst Appl 38(5):4198–4205CrossRefGoogle Scholar
  10. Deng W, Chen R, Gao J, Song YJ, Xu JJ (2012) A novel parallel hybrid intelligence optimization algorithm for a function approximation problem. Comput Math Appl 63(1):325–336CrossRefMATHMathSciNetGoogle Scholar
  11. Fan SKS, Liang YC, Zahara E (2004) Hybrid simplex search and particle swarm optimization for the global optimization for multimodal functions. Eng Optim 36(5):401–418CrossRefGoogle Scholar
  12. Goldberg DE (1989) Genetic algorithm in search optimization and machine learning. Addison-Wesley, New YorkGoogle Scholar
  13. Gori M, Tesi A (1992) On the problem of local minima in back-propagation. IEEE Trans Pattern Anal 14(1):76–86CrossRefGoogle Scholar
  14. Holland J (1975) Adaptation in natural and artificial systems. The University of Michigan Press, Ann ArborGoogle Scholar
  15. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(7):359–366CrossRefGoogle Scholar
  16. Hsu CC, Gao CH (2008) Particle swarm optimization incorporating simplex search and center particle for global optimization. In: Proceedings of the IEEE conference on soft computing in industrial applications, pp 25–27Google Scholar
  17. Juang CF (2004) A hybrid of genetic algorithm and particle swarm optimization for recurrent network design. IEEE Trans Syst Man Cybern Part B Cybern 34(3):997–1006CrossRefMathSciNetGoogle Scholar
  18. Juang CF, Chung IF, Hsu CH (2007) Automatic construction of feedforward/recurrent fuzzy systems by clustering-aided simplex particle swarm optimization. Fuzzy Set Syst 158(18):1979–1996CrossRefMATHMathSciNetGoogle Scholar
  19. Kennedy J, Eberhart RC (1995) Particle swarm optimization. In: Proceedings of IEEE international conference on neural networks, pp 1942–1984Google Scholar
  20. Liu A, Zahaya E, Yang MT (2012) A modified NM-PSO method for parameter estimation problems of models. J Appl Math, p 2012Google Scholar
  21. Mendes R, Cortez P, Rocha M, Neves J (2002) Particle swarms for feedforward neural network training. In: Proceedings of the international joint conference on neural networks, pp 1895–1899 Google Scholar
  22. Maindonald J, John Braun W (2012) DAAG: Data Analysis And Graphics data and functions. http://CRAN.R-project.org/package=DAAG
  23. Nelder JA, Mead R (1965) A simplex method for function minimization. Comput J 7:308–313Google Scholar
  24. Nazareth L, Tseng P (2002) Gilding the lily: a variant of the Nelder–Mead algorithm based on golden-section search. Comput Optim Appl 22(1):133–144CrossRefMATHMathSciNetGoogle Scholar
  25. Rusiecki A (2007) Robust LTS backpropagation learning algorithm. Lect Notes Comput Sci 4507:102–109CrossRefGoogle Scholar
  26. R Development Core Team (2011) R: A Language and Environment for Statistical Computing. http://www.R-project.org/
  27. Shaw S, Kinsner W (1996) Chaotic simulated annealing in multilayer feedforward networks. In: Proceedings of Canadian conference on electrical and computer engineering, pp 265–269Google Scholar
  28. Salerno J (1997) Using the particle swarm optimization technique to train a recurrent neural model. In: Proceedings of ninth IEEE international conference tools with artificial intelligence, pp 45–49Google Scholar
  29. Trelea IC (2003) The particle swarm optimization algorithm: Convergence analysis and parameter selection. Inform Process Lett 85:317–325CrossRefMATHMathSciNetGoogle Scholar
  30. Tan Y, Ruan G (2010) A three-layer back-propagation neural network for spam detection using artificial immune concentration. Soft Comput 14(3):139–150Google Scholar
  31. Venkatesh YV, Kumar Raja S (2006) Multiple contour extraction from graylevel images using an artificial neural network. IEEE Trans Image Process 15(5):892–899CrossRefGoogle Scholar
  32. Yu J, Xi L, Wang S (2007) An improved particle swarm optimization for evolving feedforward artificial neural networks. Neural Process Lett 26(4):217–231CrossRefGoogle Scholar
  33. Zhang C, Shao H, Li Y (2000) Particle swarm optimization for evolving artificial neural network. In: Proceedings of the IEEE international conference on system, man, and cybernetics, pp 2487–2490Google Scholar
  34. Zhang JR, Zhang J, Lok TM, Lyu MR (2007) A hybrid particle swarm optimization back-propagation algorithm for feedforward neural network training. Appl Math Comput 185(3):1026–1037CrossRefMATHGoogle Scholar
  35. Zahaya E, Liu A (2010) Solving parameter identification problem by hybrid particle swarm optimization. In: Proceedings of the international multiconference of engineers and computer scientists, pp 36–38Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Shih-Hui Liao
    • 1
  • Jer-Guang Hsieh
    • 2
  • Jyh-Yeong Chang
    • 1
  • Chin-Teng Lin
    • 1
  1. 1.Department of Electrical and Control EngineeringNational Chiao Tung UniversityHsinchuTaiwan, ROC
  2. 2.Department of Electrical EngineeringI-Shou UniversityKaohsiung CountyTaiwan

Personalised recommendations