An Accelerated Particle Swarm Optimization Based Levenberg Marquardt Back Propagation Algorithm

  • Nazri Mohd Nawi
  • Abdullah khan
  • M. Z. Rehman
  • Maslina Abdul Aziz
  • Tutut Herawan
  • Jemal H. Abawajy
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8835)


The Levenberg Marquardt (LM) algorithm is one of the most effective algorithms in speeding up the convergence rate of the Artificial Neural Networks (ANN) with Multilayer Perceptron (MLP) architectures. However, the LM algorithm suffers the problem of local minimum entrapment. Therefore, we introduce several improvements to the Levenberg Marquardt algorithm by training the ANNs with meta-heuristic nature inspired algorithm. This paper proposes a hybrid technique Accelerated Particle Swarm Optimization using Levenberg Marquardt (APSO_LM) to achieve faster convergence rate and to avoid local minima problem. These techniques are chosen since they provide faster training for solving pattern recognition problems using the numerical optimization technique.The performances of the proposed algorithm is evaluated using some bench mark of classification’s datasets. The results are compared with Artificial Bee Colony (ABC) Algorithm using Back Propagation Neural Network (BPNN) algorithm and other hybrid variants.Based on the experimental result, the proposed algorithms APSO_LM successfully demonstrated better performance as compared to other existing algorithms in terms of convergence speed and Mean Squared Error (MSE) by introducing the error and accuracy in network convergence.


Artificial Neural Networks Particle Swarm Optimization Levenberg Marquardt Back Propagation Meta-heuristic optimization Nature inspired algorithms 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Zheng, H., Meng, W., Gong, B.: Neural Network and its Application on Machine fault Diagnosis. In: ICSYSE 1992, pp. 576–579 (1992)Google Scholar
  2. 2.
    Kosko, B.: Neural Network and Fuzzy Systems, 1st edn. Prentice Hall of India (1992)Google Scholar
  3. 3.
    Basheer, I.A., Hajmeer: Artificial Neural Networks: fundamentals, computing, design and application. Journal of Microbiological Methods 43(1), 3–31 (2000)CrossRefGoogle Scholar
  4. 4.
    Krasnopolsky, V.M., Chevallier: Some Neural Network applications in environmental sciences. Part II, advancing computational efficiency of environmental numerical models. Neural Networks 16(3), 335–348 (2003)CrossRefGoogle Scholar
  5. 5.
    Coppin, B.: Artificial Intelligence Illuminated. Jones and Bartlet illuminated Series, USA, pp. 291–324 (2004)Google Scholar
  6. 6.
    Nawi, N.M., Khan, A., Rehman, M.Z.: A new cuckoo search based levenberg-marquardt (CSLM) algorithm. In: Murgante, B., Misra, S., Carlini, M., Torre, C.M., Nguyen, H.-Q., Taniar, D., Apduhan, B.O., Gervasi, O. (eds.) ICCSA 2013, Part I. LNCS, vol. 7971, pp. 438–451. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  7. 7.
    Nawi, N.M., Khan, A., Rehman, M.Z.: A new back-propagation neural network optimized with cuckoo search algorithm. In: Murgante, B., Misra, S., Carlini, M., Torre, C.M., Nguyen, H.-Q., Taniar, D., Apduhan, B.O., Gervasi, O. (eds.) ICCSA 2013, Part I. LNCS, vol. 7971, pp. 413–426. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  8. 8.
    Contreras, J., et al.: ARIMA models to predict next-day electricity prices. IEEE Transactions on Power Systems 18, 1014–1020 (2003)CrossRefGoogle Scholar
  9. 9.
    Leung, C.T., Chow, T.: A hybrid global learning algorithm based on global search and least squares techniques for back propagation networks. In: International Conference on Neural Networks, pp. 1890–1895 (1997)Google Scholar
  10. 10.
    Zhang, J.-R., et al.: A hybrid particle swarm optimization–back-propagation algorithm for feedforward neural network training. Applied Mathematics and Computation 185, 1026–1037 (2007)CrossRefzbMATHGoogle Scholar
  11. 11.
    Ahmed, W.A.M., Saad, E.S.M., Aziz, E.S.: Modified Back Propagation Algorithm for Learning Artificial Neural Networks. In: Proceedings of the Eighteenth National Radio Science Conference, pp. 345–352 (2001)Google Scholar
  12. 12.
    Nawi, N.M., et al.: Countering the Problem of Oscillations in Bat-BP Gradient Trajectory by Using Momentum. In: Proceedings of the First International Conference on Advanced Data and Information Engineering (DaEng 2013), pp. 103–110. Springer Singapore (2014)Google Scholar
  13. 13.
    Jin, W., et al.: The improvements of BP neural network learning algorithm. In: Proceedings of 5th International Conference on Signal Processing WCCC-ICSP 2000, vol. 3, pp. 1647–1649 (2000)Google Scholar
  14. 14.
    Nawi, N.M., Ransing, R.S., Salleh, M.N.M., Ghazali, R., Hamid, N.A.: An Improved Back Propagation Neural Network Algorithm on Classification Problems. In: Zhang, Y., Cuzzocrea, A., Ma, J., Chung, K.-i., Arslan, T., Song, X. (eds.) DTA and BSBT 2010. CCIS, vol. 118, pp. 177–188. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Mohd Nawi, N., et al.: BPGD-AG: A New Improvement Of Back-Propagation Neural Network Learning Algorithms With Adaptive Gain. Journal of Science and Technology 2 (2011)Google Scholar
  16. 16.
    Nawi, N.M., Ghazali, R., Salleh, M.N.M.: The Development of Improved Back-Propagation Neural Networks Algorithm for Predicting Patients with Heart Disease. In: Zhu, R., Zhang, Y., Liu, B., Liu, C. (eds.) ICICA 2010. LNCS, vol. 6377, pp. 317–324. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  17. 17.
    Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 5, 989–993 (1994)CrossRefGoogle Scholar
  18. 18.
    Wilamowski, B., et al.: Neural network trainer with second order learning algorithms. In: 11th International Conference on Intelligent Engineering Systems, INES 2007, pp. 127–132 (2007)Google Scholar
  19. 19.
    Lourakis, M.I.: A brief description of the Levenberg-Marquardt algorithm implemented by levmar (2005)Google Scholar
  20. 20.
    Xue, Q., et al.: Improved LMBP algorithm in the analysis and application of simulation data. In: 2010 International Conference on Computer Application and System Modeling (ICCASM), vol. 6, pp. 545–547 (2010)Google Scholar
  21. 21.
    Yan, J., et al.: Levenberg-Marquardt algorithm applied to forecast the ice conditions in Ningmeng Reach of the Yellow River. In: Fifth International Conference on Natural Computation, ICNC 2009, pp. 184–188 (2009)Google Scholar
  22. 22.
    Ozturk, C., Karaboga, D.: Hybrid artificial bee colony algorithm for neural network training. In: 2011 IEEE Congress on Evolutionary Computation (CEC), pp. 84–88 (2011)Google Scholar
  23. 23.
    Nawi, A.K.N.M., Rehman, M.Z.: A New Levenberg-Marquardt based Back-propagation Algorithm trained with Cuckoo Search. In: ICEEI 2013 Procedia Technology 8C, pp. 18–24 (2013)Google Scholar
  24. 24.
    Bäck, T., Schwefel, H.-P.: An overview of evolutionary algorithms for parameter optimization. Evolutionary Computation 1, 1–23 (1993)CrossRefGoogle Scholar
  25. 25.
    Blum, C., Socha, K.: Training feed-forward neural networks with ant colony optimization: An application to pattern classification. In: Fifth International Conference on Hybrid Intelligent Systems, HIS 2005, p. 6 (2005)Google Scholar
  26. 26.
    Karaboga, D., Akay, B., Ozturk, C.: Artificial bee colony (ABC) optimization algorithm for training feed-forward neural networks. In: Torra, V., Narukawa, Y., Yoshida, Y. (eds.) MDAI 2007. LNCS (LNAI), vol. 4617, pp. 318–329. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  27. 27.
    Montana, D.J., Davis, L.: Training Feedforward Neural Networks Using Genetic Algorithms. In: IJCAI, pp. 762–767 (1989)Google Scholar
  28. 28.
    Nawi, N.M., Rehman, M.Z., Khan, A.: A New Bat Based Back-Propagation (BAT-BP) Algorithm. In: Swiątek, J., Grzech, A., Swiątek, P., Tomczak, J.M. (eds.) Advances in Systems Science. AISC, vol. 240, pp. 395–404. Springer, Heidelberg (2014)Google Scholar
  29. 29.
    Zhang, J.Z.T.L.J., Lyu, M.: A hybrid particle swarm optimization back propagation algorithm for neural network training. Applied Mathematics and Computation 185, 1026–1037 (2007)CrossRefzbMATHGoogle Scholar
  30. 30.
    Yang, X.-S.: Nature-inspired metaheuristic algorithms. Luniver Press (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Nazri Mohd Nawi
    • 1
  • Abdullah khan
    • 1
  • M. Z. Rehman
    • 1
  • Maslina Abdul Aziz
    • 2
  • Tutut Herawan
    • 3
    • 4
  • Jemal H. Abawajy
    • 2
  1. 1.Faculty of Computer Science and Information TechnologyUniversiti Tun Hussein Onn MalaysiaParit RajaMalaysia
  2. 2.School of Information TechnologyDeakin UniversityGeelongAustralia
  3. 3.University of MalayaKuala LumpurMalaysia
  4. 4.AMCS Research CenterYogyakartaIndonesia

Personalised recommendations