The Parallel Modification to the Levenberg-Marquardt Algorithm

  • Jarosław BilskiEmail author
  • Bartosz Kowalczyk
  • Konrad Grzanek
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10841)


The paper presents a parallel approach to the Levenberg-Marquardt algorithm (also called LM or LMA). The first section contains the mathematical basics of the classic LMA. Then the parallel modification to LMA is introduced. The classic Levenberg-Marquardt algorithm is sufficient for a training of small neural networks. For bigger networks the algorithm complexity becomes too big for the effective teaching. The main scope of this paper is to propose more complexity efficient approach to LMA by parallel computation. The proposed modification to LMA has been tested on a few function approximation problems and has been compared to the classic LMA. The paper concludes with the resolution that the parallel modification to LMA could significantly improve algorithm performance for bigger networks. Summary also contains a several proposals for the possible future work directions in the considered area.


Feed-forward neural network Parallel neural network training algorithm Optimization problem Levenberg-Marquardt algorithm QR decomposition Givens rotation 


  1. 1.
    Starczewski, A.: A new validity index for crisp clusters. Pattern Anal. Appl. 20(3), 687–700 (2015)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Starczewski, A., Krzyżak, A.: Improvement of the validity index for determination of an appropriate data partitioning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10246, pp. 159–170. Springer, Cham (2017). Scholar
  3. 3.
    Bilski, J., Wilamowski, B.M.: Parallel Levenberg-Marquardt algorithm without error backpropagation. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10245, pp. 25–39. Springer, Cham (2017). Scholar
  4. 4.
    Bilski, J., Kowalczyk, B., Żurada, J.M.: Parallel implementation of the givens rotations in the neural network learning algorithm. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10245, pp. 14–24. Springer, Cham (2017). Scholar
  5. 5.
    Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent RTRN neural network learning. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 11–16. Springer, Heidelberg (2008). Scholar
  6. 6.
    Bilski, J., Smoląg, J.: Parallel architectures for learning the rtrn and elman dynamic neural network. IEEE Trans. Parallel Distrib. Syst. 26(9), 2561–2570 (2015)CrossRefGoogle Scholar
  7. 7.
    Bilski, J., Smoląg, J., Żurada, J.M.: Parallel approach to the Levenberg-Marquardt learning algorithm for feedforward neural networks. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds.) ICAISC 2015. LNCS (LNAI), vol. 9119, pp. 3–14. Springer, Cham (2015). Scholar
  8. 8.
    Marqardt, D.: An algorithm for last-sqares estimation of nonlinear paeameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963)CrossRefGoogle Scholar
  9. 9.
    Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Netw. 5(6), 989–993 (1994)CrossRefGoogle Scholar
  10. 10.
    Werbos, J.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Harvard University, Cambridge (1974)Google Scholar
  11. 11.
    Cpałka, K., Łapa, K., Przybył, A.: A new approach to design of control systems using genetic programming. Inf. Technol. Control 44(4), 433–442 (2015)Google Scholar
  12. 12.
    Łapa, K., Cpałka, K.: On the application of a hybrid genetic-firework algorithm for controllers structure and parameters selection. In: Borzemski, L., Grzech, A., Świątek, J., Wilimowska, Z. (eds.) Information Systems Architecture and Technology: Proceedings of 36th International Conference on Information Systems Architecture and Technology – ISAT 2015 – Part I. AISC, vol. 429, pp. 111–123. Springer, Cham (2016). Scholar
  13. 13.
    Łapa, K., Cpałka, K., Galushkin, A.I.: A new interpretability criteria for neuro-fuzzy systems for nonlinear classification. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds.) ICAISC 2015. LNCS (LNAI), vol. 9119, pp. 448–468. Springer, Cham (2015). Scholar
  14. 14.
    Khan, N.A., Shaikh, A.: A smart amalgamation of spectral neural algorithm for nonlinear Lane-Emden equations with simulated annealing. J. Artif. Intell. Soft Comput. Res. 7(3), 215–224 (2017)CrossRefGoogle Scholar
  15. 15.
    Liu, H., Gegov, A., Cocea, M.: Rule based networks: an efficient and interpretable representation of computational models. J. Artif. Intell. Soft Comput. Res. 7(2), 111–123 (2017)CrossRefGoogle Scholar
  16. 16.
    Notomista, G., Botsch, M.: A machine learning approach for the segmentation of driving Maneuvers and its application in autonomous parking. J. Artif. Intell. Soft Comput. Res. 7(4), 243–255 (2017)CrossRefGoogle Scholar
  17. 17.
    Rotar, C., Lantovics, L.B.: Directed evolution - a new Metaheuristc for optimization. J. Artif. Intell. Soft Comput. Res. 7(3), 183–200 (2017)CrossRefGoogle Scholar
  18. 18.
    Rutkowska, D., Nowicki, R., Hayashi, Y.: Parallel processing by implication-based neuro-fuzzy systems. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Waśniewski, J. (eds.) PPAM 2001. LNCS, vol. 2328, pp. 599–607. Springer, Heidelberg (2002). Scholar
  19. 19.
    Smoląg, J., Bilski, J.: A systolic array for fast learning of neural networks. In: V NNSC, pp. 754–758 (2000)Google Scholar
  20. 20.
    Smoląg, J., Bilski, J., Rutkowski, L.: Systolic array for neural networks. In: IV KSNiIZ, pp. 487–497 (1999)Google Scholar
  21. 21.
    Villmann, T., Bohnsack, A., Kaden, M.: Can learning vector quantization be an alternative to SVM and deep learning? Recent trends and advanced variants of learning vector quantization for classification learning. J. Artif. Intell. Soft Comput. Res. 7(1), 65–81 (2017)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Jarosław Bilski
    • 1
    Email author
  • Bartosz Kowalczyk
    • 1
  • Konrad Grzanek
    • 2
    • 3
  1. 1.Institute of Computational IntelligenceCzȩstochowa University of TechnologyCzȩstochowaPoland
  2. 2.Information Technology InstituteUniversity of Social SciencesŁódźPoland
  3. 3.Clark UniversityWorcesterUSA

Personalised recommendations