CSBPRNN: A New Hybridization Technique Using Cuckoo Search to Train Back Propagation Recurrent Neural Network

Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 285)

Abstract

Nature inspired meta-heuristic algorithms provide derivative-free solution to optimize complex problems. Cuckoo Search (CS) algorithm is one of the most modern addition to the group of nature inspired optimization meta-heuristics. The Simple Recurrent Networks (SRN) were initially trained by Elman with the standard back propagation (SBP) learning algorithm which is less capable and often takes enormous amount of time to train a network of even a moderate size. And the complex error surface of the SBP makes many training algorithms are prone to being trapped in local minima. This paper proposed a new meta-heuristic based Cuckoo Search Back Propagation Recurrent Neural Network (CSBPRNN) algorithm. The CSBPRNN is based on Cuckoo Search to train BPRNN in order to achieve fast convergence rate and to avoid local minima problem. The performance of the proposed CSBPRNN is compared with Artificial Bee Colony using BP algorithm, and other hybrid variants. Specifically OR and XOR datasets are used. The simulation results show that the computational efficiency of BP training process is highly enhanced when coupled with the proposed hybrid method.

Keywords

Back propagation Cuckoo search Local minima Artificial bee colony algorithm Meta-heuristic optimization Recurrent neural network 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgments

The Authors would like to thank Office of Research, Innovation, Commercialization and Consultancy Office (ORICC), Universiti Tun Hussein Onn Malaysia (UTHM) and Ministry of Higher Education (MOHE) Malaysia for financially supporting this Research under Fundamental Research Grant Scheme (FRGS) vote no. 1236.

References:

  1. 1.
    Elman, J. L.: Finding structure in time, J. Cognitive Science, Vol.14 (2), pp. 179–211, (1990)Google Scholar
  2. 2.
    Barbounis, T.G., Theocharis, J.B., et al., Long-term wind speed and power forecasting using local recurrent neural network models, J. IEEE Transactions on Energy Conversion, vol. 21,(1,) pp. 273–284, (2006)Google Scholar
  3. 3.
    Goedtel, A., Dasilva, I.N., and Amaral Semi, P.J., Recurrent Neural Network for Induction Motor Speed Estimation in Industry Application, In: IEEE MELECON, pp. 1134–1137, August (2006)Google Scholar
  4. 4.
    Xiao, P., Venayagamoorthy, G. K., and Corzine, K. A.: Combined Training of Recurrent Neural Networks with Particle Swarm Optimization and Back propagation Algorithms for Impedance Identification, In: Proceedings of the IEEE Swarm Intelligence Symposium (2007)Google Scholar
  5. 5.
    Hecht-Nielsen, R.: Neurocomputing, Addison-Wesley, Reading, PA, (1990)Google Scholar
  6. 6.
    Giles, C. L., Lawrence, S., and Tsoi, A. C.: Rule inference for financial prediction using recurrent neural networks, In: IEEE Conference on Computational Intelligence for Financial Engineering, IEEE Press. (1997)Google Scholar
  7. 7.
    Li, S., Wunsch II, D. C., O’Hair, E., and Giesselmann, M. G.: Wind turbine power estimation by neural networks with Kalman filter training on a SIMD parallel machine, In: International Joint Conference on Neural Networks, (1999)Google Scholar
  8. 8.
    Coulibay, P., Anctil, F., and Rousselle, J.: Real-time short-term water inflows forecasting using recurrent neural networks, In: International Joint Conference on Neural Networks, (1999)Google Scholar
  9. 9.
    Gudise, V.G. and Venayagamoorthy, G.K.: Comparison of particle swarm optimization and backpropagation as training algorithms for neural networks, In: IEEE Swarm Intelligence Symposium, pp. 110–117. April (2003)Google Scholar
  10. 10.
    Janson, D.J. and Frenzel, J.F.: Training product unit neural networks with genetic algorithms, J. IEEE Intelligent Systems and Their Applications, vol. 8 (5), pp. 26–33, Oct. (1993)Google Scholar
  11. 11.
    Salerno, J., Using the particle swarm optimization technique to train a recurrent neural model, In: Ninth IEEE International Conference on Tools with Artificial Intelligence, pp. 45–49, Nov. (1997)Google Scholar
  12. 12.
    Werbos, P.: Back propagation through time: what it does and how to do it, In: Proceedings of the IEEE, vol. 78 (10), pp. 1550–1560. (1990)Google Scholar
  13. 13.
    Temurtas, F., Yumusak, N., Gunturkun, R., Temurtas, H., and Cerezci, O.: Elman’s Recurrent Neural Networks Using Resilient Back Propagation for Harmonic Detection, In: 8th Pacific Rim International Conference on Artificial Intelligent Proceedings, vol. 3157, pp. 422–428. (2004)Google Scholar
  14. 14.
    Rumelhart D.E. Hinton G.E., and Williams R.J.: Learning Representations by Back– Propagating Errors Nature, vol. 323, pp. 533-536. (1986)Google Scholar
  15. 15.
    Nawi, N. M., Rehman, M. Z., and Khan, A.: A New Bat-Based Back-propagation (BAT-BP) Algorithm, In: ICSS 2013, Wrocław, Poland, September 10-12. (2013)Google Scholar
  16. 16.
    Nawi, N. M., Khan, A., and Rehman, M. Z.: A New Back-propagation Neural Network optimized with Cuckoo Search Algorithm, In: ICCSA-2013, pp. 413–426. (2013)Google Scholar
  17. 17.
    Nawi, N. M., Khan, A., and Rehman, M. Z.: A New Cuckoo Search based Levenberg-Marquardt (CSLM) Algorithm, In: ICCSA-2013, pp. 438–451. (2013)Google Scholar
  18. 18.
    Nawi, N. M., Khan, A., and Rehman, M. Z.: A New Levenberg-Marquardt based Back-propagation Algorithm trained with Cuckoo Search, In: ICEEI-2013, pp.18–24. (2013)Google Scholar
  19. 19.
    Lahmiri, S.: A comparative study of backpropagation algorithms in financial prediction. J. International Journal of Computer Science, Engineering and Applications (IJCSEA), vol.1 (4). (2011)Google Scholar
  20. 20.
    Nawi, N. M., Ransing, R. S., Salleh, M.N.M., Ghazali,R., and Hamid, N.A, An improved back propagation neural network algorithm on classification problems, J. Communications in Computer and Information Science vol. 118, pp. 177–188. (2010)Google Scholar
  21. 21.
    Nawi, N.M., Ghazali, R., Salleh, M.N.M. The development of improved back-propagation neural networks algorithm for predicting patients with heart disease, J. LNCS, vol. 6377 (4), pp. 317–324. (2010)Google Scholar
  22. 22.
    Yang. XS., and Deb. S.: Cuckoo search via Lévy flights, In: World Congress on Nature & Biologically Inspired Computing, India, pp. 210–214. (2009)Google Scholar
  23. 23.
    Yang, X. S., and Deb, S.: Engineering Optimisation by Cuckoo Search, Int. J. of Mathematical Modelling and Numerical Optimisation, vol. 1 (4), pp. 330– 343. (2010)Google Scholar
  24. 24.
    Tuba, M., Subotic, M., Stanarevic, N.: Modified cuckoo search algorithm for unconstrained optimization problems, In: European Computing Conference, pp. 263–268. (2011)Google Scholar
  25. 25.
    Tuba, M., Subotic, M., Stanarevic, N.: Performance of a Modified Cuckoo Search Algorithm for Unconstrained Optimization Problems, J. Faculty of Computer Science, vol. 11 (2), pp. 62–74. (2012)Google Scholar
  26. 26.
    Chaowanawate, K., and Heednacram, A.: Implementation of Cuckoo Search in RBF Neural Network for Flood Forecasting, In: Fourth International Conference on Computational Intelligence, Communication Systems and Networks, pp. 22–26. (2012)Google Scholar
  27. 27.
    Pavlyukevich, I.: Levy flights, non-local search and simulated annealing, J. Journal of Computational Physics, vol. 226 (2), pp. 1830–1844. (2007)Google Scholar
  28. 28.
    Walton, S., Hassan, O., Morgan, K., and Brown, M.: Modified cuckoo search: A new gradient free optimisation algorithm. J. Chaos, Solitons& Fractals, vol. 44 (9), pp. 710–718. (2011)Google Scholar
  29. 29.
    Williams, R. J. and Peng, J.: An efficient gradient-based algorithm for on-line training ofrecurrent network trajectories, J. Neural Computation, vol.2, pp. 490–501. (1990)Google Scholar
  30. 30.
    Nandy, S., Sarkar, P. P., and Das, A.: Training a Feed-forward Neural Network with Artificial Bee Colony Based Backpropagation Method, J. International Journal of Computer Science & Information Technology (IJCSIT), vol. 4 (4), pp. 33–46. (2012)Google Scholar
  31. 31.
    Ozturk, C., and Karaboga, D.: Hybrid Artificial Bee Colony algorithm for neural network training, In: IEEE Congress of Evolutionary Computation (CEC), pp. 84–88. (2011)Google Scholar
  32. 32.
    Karaboga, D., and Ozturk, C..: Neural networks training by artificial bee colony algorithm on pattern classification, J. Neural Network World, vol. 19, no. 10, pp. 279–292, (2009).Google Scholar

Copyright information

© Springer Science+Business Media Singapore 2014

Authors and Affiliations

  • Nazri Mohd. Nawi
    • 1
  • Abdullah khan
    • 1
  • M. Z. Rehman
    • 1
  1. 1.Software and Multimedia Centre, Faculty of Computer Science and Information TechnologyUniversiti Tun Hussein Onn Malaysia (UTHM)BatuPahatMalaysia

Personalised recommendations