Abstract
The backpropagation (BP) algorithm is a worldwide used method for learning neural networks. The BP has a low computational load. Unfortunately, this method converges relatively slowly. In this paper a new approach to the backpropagation algorithm is presented. The proposed solution speeds up the BP method by using vector calculations. This modification of the BP algorithm was tested on a few standard examples. The obtained performance of both methods was compared.
This work has been supported by the Polish National Science Center under Grant 2017/27/B/ST6/02852.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, Z., Cao, J., Cai, Z., Rutkowski, L.: Anti-synchronization in fixed time for discontinuous reaction–diffusion neural networks with time-varying coefficients and time delay. IEEE Trans. Cybern. 50(6), 2758–2769 (2020). https://doi.org/10.1109/TCYB.2019.2913200
Duda, P., Rutkowski, L., Jaworski, M., Rutkowska, D.: On the parzen kernel-based probability density function learning procedures over time-varying streaming data with applications to pattern classification. IEEE Trans. Cybern. 50(4), 1683–1696 (2020). https://doi.org/10.1109/TCYB.2018.2877611
Lin, L., Cao, J., Rutkowski, L.: Robust event-triggered control invariance of probabilistic boolean control networks. IEEE Trans. Neural Netw. Learn. Syst. 31(3), 1060–1065 (2020). https://doi.org/10.1109/TNNLS.2019.2917753
Liu, Y., Zheng, Y., Lu, J., Cao, J., Rutkowski, L.: Constrained quaternion-variable convex optimization: a quaternion-valued recurrent neural network approach. IEEE Trans. Neural Netw. Learn. Syst. 31(3), 1022–1035 (2020). https://doi.org/10.1109/TNNLS.2019.2916597
Gabryel, M., Przybyszewski, K.: The dynamically modified BoW algorithm used in assessing clicks in online ads. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019, Part II. LNCS (LNAI), vol. 11509, pp. 350–360. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20915-5_32
Gabryel, M.: The bag-of-words method with different types of image features and dictionary analysis. J. Univ. Comput. Sci. 24(4), 357–371 (2018)
Starczewski, A.: A new validity index for crisp clusters. Pattern Anal. Appl. 20(3), 687–700 (2015). https://doi.org/10.1007/s10044-015-0525-8
Starczewski, A., Goetzen, P., Er, M.J.: A new method for automatic determining of the DBSCAN parameters. J. Artif. Intell. Soft Comput. Res. 10(3), 209–221 (2019)
Łapa, K., Cpałka, K., Wang, L.: New method for design of fuzzy systems for nonlinear modelling using different criteria of interpretability. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014, Part I. LNCS (LNAI), vol. 8467, pp. 217–232. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07173-2_20
Zalasiński, M., Cpałka, K., Er, M.J.: New method for dynamic signature verification using hybrid partitioning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014, Part II. LNCS (LNAI), vol. 8468, pp. 216–230. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07176-3_20
Szczypta, J., Przybył, A., Cpałka, K.: Some aspects of evolutionary designing optimal controllers. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013, Part II. LNCS (LNAI), vol. 7895, pp. 91–100. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38610-7_9
Rutkowski, T., Romanowski, J., Woldan, P., Staszewski, P., Nielek, R., Rutkowski, L.: A content-based recommendation system using neuro-fuzzy approach. In: 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–8 (2018)
Rutkowski, T., Łapa, K., Nowicki, R., Nielek, R., Grzanek, K.: On explainable recommender systems based on fuzzy rule generation techniques. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019, Part I. LNCS (LNAI), vol. 11508, pp. 358–372. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20912-4_34
Rutkowski, L.: Computational Intelligence. Methods and Techniques. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-76288-1
Liao, J., Liu, T., Liu, M., Wang, J., Wang, Y., Sun, H.: Multi-context integrated deep neural network model for next location prediction. IEEE Access 6, 21980–21990 (2018)
Zalasiński, M., Łapa, K., Cpałka, K., Przybyszewski, K., Yen, G.G.: On-line signature partitioning using a population based algorithm. J. Artif. Intell. Soft Comput. Res. 10(1), 5–13 (2020)
Nobukawa, S., Nishimura, H., Yamanishi, T.: Pattern classification by spiking neural networks combining self-organized and reward-related spike-timing-dependent plasticity. J. Artif. Intell. Soft Comput. Res. 9(4), 283–291 (2019)
de Souza, G.B., da Silva Santos, D.F., Gonçalves Pires, R., Nilceu Marananil, A., Paulo Papa, J.: Deep features extraction for robust fingerprint spoofing attack detection. J. Artif. Intell. Soft Comput. Res. 9(1), 41–49 (2019)
Bilski, J.: The UD RLS algorithm for training the feedforward neural networks. Int. J. Appl. Math. Comput. Sci. 15(1), 101–109 (2005)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Internal Representations by Error Propagation, Parallel Distributed Processing, vol. 1. The MIT Press, Cambridge (1986).Ch. 8, Rumelhart dE. and McCelland J. (red.)
Wilamowski, B.M., Yo, H.: Neural network learning without backpropagation. IEEE Trans. Neural Netw. 21(11), 1793–1803 (2010)
Żurada, J.: Introduction to Artificial Neural Systems. West Publishing Co., St. Paul (1992)
Grycuk, R., Najgebauer, P., Kordos, M., Scherer, M.M., Marchlewska, A.: On the topological properties of the certain neural networks. J. Artif. Intell. Soft Comput. Res. 10(2), 257–268 (2018)
Shewalkar, A., Nyavanandi, D., Ludwig, S.A.: Fast image index for database management engines. J. Artif. Intell. Soft Comput. Res. 9(4), 95–111 (2020)
Scherer, R.: Computer Vision Methods for Fast Image Classification and Retrieval. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-12195-2
Bilski, J., Litwiński, S., Smola̧g, J.: Parallel realisation of QR algorithm for neural networks learning. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 158–165. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24844-6_19
Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent RTRN neural network learning. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 11–16. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-69731-2_2
Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent elman neural network learning. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010, Part II. LNCS (LNAI), vol. 6114, pp. 19–25. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13232-2_3
Bilski, J., Smoląg, J.: Parallel realisation of the recurrent multi layer perceptron learning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS (LNAI), vol. 7267, pp. 12–20. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29347-4_2
Bilski, J., Smoląg, J., Żurada, J.M.: Parallel approach to the levenberg-marquardt learning algorithm for feedforward neural networks. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2015, Part I. LNCS (LNAI), vol. 9119, pp. 3–14. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19324-3_1
Bilski, J., Wilamowski, B.M.: Parallel levenberg-marquardt algorithm without error backpropagation. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017, Part I. LNCS (LNAI), vol. 10245, pp. 25–39. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59063-9_3
Smoląg, J., Bilski, J.: A systolic array for fast learning of neural networks. In: Proceedings of V Conference Neural Networks and Soft Computing, Zakopane, pp. 754–758 (2000)
Smoląg, J., Rutkowski, L., Bilski, J.: Systolic array for neural networks. In: Proceedings of IV Conference Neural Networks and Their Applications, Zakopane, pp. 487–497 (1999)
Fahlman, S.: Faster learning variations on backpropagation: an empirical study. In: Proceedings of Connectionist Models Summer School, Los Atos (1988)
Riedmiller, M., Braun, H.: A direct method for faster backpropagation learning: the RPROP algorithm. In: IEEE International Conference on Neural Networks, San Francisco (1993)
Werbos, J.: Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990)
Charalambous, C.: Conjugate gradient algorithm for efficient training of artificial neural networks. IEE Proc. G - Circ. Devices Syst. 139(3), 301–310 (1992)
Fletcher, R., Powell, M.J.D.: A rapidly convergent descent method for minimization. Comput. J. 6, 163–168 (1963)
Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradients. Comput. J. 7, 149–154 (1964)
Nocedal, J., Wright, S.J.: Conjugate Gradient Methods in Numerical Optimization, pp. 497–528. Springer, New York (2006)
Polak, E.: Computational Methods in Optimization: A Unified Approach. Academic Press, New York (1971)
Jin, X.-B., Zhang, X.-Y., Huang, K., Geng, G.-G.: Stochastic conjugate gradient algorithm with variance reduction. IEEE Trans. Neural Netw. Learn. Syst. 30(5), 1360–1369 (2019)
Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Netw. 5(6), 989–993 (1994)
Bilski, J., Kowalczyk, B., Marchlewska, A., Zurada, J.M.: Local levenberg-marquardt algorithm for learning feedforwad neural networks. J. Artif. Intell. Soft Comput. Res. 10(4), 299–316 (2020). https://doi.org/10.2478/jaiscr-2020-0020
Rafajłowicz, E., Rafajłowicz, W.: Iterative learning in optimal control of linear dynamic processes. Int. J. Control 91(7), 1522–1540 (2018)
Rafajłowicz, E., Rafajłowicz, W.: Iterative learning in repetitive optimal control of linear dynamic processes. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2016, Part I. LNCS (LNAI), vol. 9692, pp. 705–717. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39378-0_60
Jurewicz, P., Rafajłowicz, W., Reiner, J., Rafajłowicz, E.: Simulations for tuning a laser power control system of the cladding process. In: Saeed, K., Homenda, W. (eds.) CISIM 2016. LNCS, vol. 9842, pp. 218–229. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45378-1_20
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Bilski, J., Smoląg, J., Najgebauer, P. (2021). Modification of Learning Feedforward Neural Networks with the BP Method. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2021. Lecture Notes in Computer Science(), vol 12854. Springer, Cham. https://doi.org/10.1007/978-3-030-87986-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-87986-0_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87985-3
Online ISBN: 978-3-030-87986-0
eBook Packages: Computer ScienceComputer Science (R0)