Abstract
This paper gives a formal convergence analysis on the mini-batch training algorithm for noise resilient radial basis function (RBF) networks. Unlike the conventional analysis which assumes that the mini-batch process is operated in a stochastic manner, we consider that the mini-batch training process is operated in a deterministic manner. The deterministic process divides the training samples into a number of fixed mini-batches, and the mini-batches are presented in a fixed order. This paper first states the noise resilient objective function for weight noise and weight fault. We then derive the mini-batch training algorithm for this noise resilient objective function. Our main contribution is the convergence analysis on the mini-batch training algorithm. We show that under the deterministic setting, the mini-batch training algorithm converges. The converged weight vector is asymptotically close to the optimal batch mode solution. Also, we derive the sufficient conditions (the learning rate range) for convergence. Our theoretical results can be applied to not only the noise resilient objective function but also a large class of objective functions.
Similar content being viewed by others
Notes
In this case, the numbers \(\kappa _i\)’s of samples in mini-batches should be greater than the number M of RBF nodes.
Note that to the best of knowledge, there is not other mini-batch algorithms for the noise resilient issue.
References
Movassagh AA, Alzubi JA, Gheisari M, Rahimi M, Mohan S, Abbasi AA, Nabipour N (2021) Artificial neural networks training algorithm integrating invasive weed optimization with differential evolutionary model, J Ambient Intell Human Comput 1–9
Soni B, Mathur P, Bora A (2021) In depth analysis, applications and future issues of artificial neural network. In: Enabling AI applications in data science, Springer, pp 149–183
Mhara MAOA (2021) Complexity neural networks for estimating flood process in internet-of-things empowered smart city, Available at SSRN 3775433
Gheisari M, Najafabadi HE, Alzubi JA, Gao J, Wang G, Abbasi AA, Castiglione A (2021) Obpp: An ontology-based framework for privacy-preserving in iot-based smart city. Fut Generation Comput Syst 123:1–13
Chandrasekaran K, Selvaraj J, Amaladoss CR, Veerapan L (2021) Hybrid renewable energy based smart grid system for reactive power management and voltage profile enhancement using artificial neural network, Energy Sources, Part A: Recovery, Utilization, and Environmental Effects, 1–24
Alzubi OA, Nazir J, Hamdoun H (2015) Cyber attack challenges and resilience for smart grids, Euro J Sci Res
Abukharis S, Alzubi JA, Alzubi OA, Alamri S (2014) Packet error rate performance of ieee802. 11g under bluetooth interface. Res J Appl Sci Eng Technol 8(12):1419–1423
Chan Y-C, Wong EW, Leung CS (2021) Evaluating non-hierarchical overflow loss systems using teletraffic theory and neural networks. IEEE Commun Lett 25(5):1486–1490
Alzubi JA (2015) Optimal classifier ensemble design based on cooperative game theory. Res J Appl Sci Eng Technol 11(12):1336–1343
Rzepecki Ł, Jaśkowski P (2021) Application of game theory against nature in supporting bid pricing in construction. Symmetry 13(1):132
Huang G, Huang G-B, Song S, You K (2015) Trends in extreme learning machines: a review. Neural Netw 61:32–48
Zhang J, Li Y, Xiao W, Zhang Z (2020) Non-iterative and fast deep learning: multilayer extreme learning machines. J Frankl Inst 357(13):8925–8955
Zhang J, Xiao W, Li Y, Zhang S (2018) Residual compensation extreme learning machine for regression. Neurocomputing 311:126–136
Leung C-S, Wan WY, Feng R (2016) A regularizer approach for RBF networks under the concurrent weight failure situation. IEEE Trans Neural Netw Learn Syst 28(6):1360–1372
Haykin S (1999) Neural networks: a comprehensive foundation, 2nd edn. Prentice Hall, Upper Saddle River, NJ, USA
Karamichailidou D, Kaloutsa V, Alexandridis A (2021) Wind turbine power curve modeling using radial basis function neural networks and tabu search. Renewable Energy 163:2137–2152
Wu H, Han Y, Geng Z, Fan J, Xu W (2022) Production capacity assessment and carbon reduction of industrial processes based on novel radial basis function integrating multi-dimensional scaling. Sustain Energy Technol Assess 49
Fei J, Wang T (2019) Adaptive fuzzy-neural-network based on rbfnn control for active power filter. Int J Mach Learn Cybernet 10(5):1139–1150
Masters D, Luschi C (2018) Revisiting small batch training for deep neural networks, arXiv preprint arXiv:1804.07612
Jin X, Sun W, Jin Z (2020) A discriminative deep association learning for facial expression recognition. Int J Mach Learn Cybernet 11(4):779–793
Cheng E-J, Chou K-P, Rajora S, Jin B-H, Tanveer M, Lin C-T, Young K-Y, Lin W-C, Prasad M (2019) Deep sparse representation classifier for facial recognition and detection system. Pattern Recognit Lett 125:71–77
Ghosh S, Pal A, Jaiswal S, Santosh K, Das N, Nasipuri M (2019) Segfast-v2: Semantic image segmentation with less parameters in deep learning for autonomous driving. Int J Mach Learn Cybernet 10(11):3145–3154
Fujiyoshi H, Hirakawa T, Yamashita T (2019) Deep learning-based image recognition for autonomous driving. IATSS Res 43(4):244–252
Wang Z, Zhou X, Wang W, Liang C (2020) Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video. Int J Mach Learn Cybernet 11(4):923–934
Si C, Tao Y, Qiu J, Lai S, Zhao J (2021) Deep reinforcement learning based home energy management system with devices operational dependencies. Int J Mach Learn Cybernet 12(6):1687–1703
Zhou Y, Wang J, Liu Y, Yan R, Ma Y (2021) Incorporating deep learning of load predictions to enhance the optimal active energy management of combined cooling, heating and power system, Energy, 121134
Wang X, Zhao Y, Pourpanah F (2020) Recent advances in deep learning
Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review, Comput Intell Neurosci 2018
Torres JM, Comesaña CI, Garcia-Nieto PJ (2019) Machine learning techniques applied to cybersecurity. Int J Mach Learn Cybernet 10(10):2823–2836
Ni D, Xiao Z, Lim MK (2019) A systematic review of the research trends of machine learning in supply chain management, Int J Mach Learn Cybernet, 1–20
Sum J, Leung C-S, Ho K (2012) Convergence analyses on on-line weight noise injection-based training algorithms for MLPs. IEEE Trans Neural Netw Learn Syst 23(11):1827–1840
Zhang H, Wu W, Liu F, Yao M (2009) Boundedness and convergence of online gradient method with penalty for feedforward neural networks. IEEE Trans Neural Netw 20(6):1050–1054
White H (1989) Some asymptotic results for learning in single hidden-layer feedforward network models. J Am Stat Assoc 84(408):1003–1013
Liu B, Kaneko T (1969) Error analysis of digital filters realized with floating-point arithmetic. Proc IEEE 57(10):1735–1747
Jeannerod C-P, Rump SM (2013) Improved error bounds for inner products in floating-point arithmetic. SIAM J Matrix Anal Appl 34(2):338–344
Diniz PS (2020) The least-mean-square (lms) algorithm, In: Adaptive Filtering, Springer, pp 61–102
Burr JB (1991) Digital neural network implementations. Neural Netw Concepts Appl Implementations 3:237–285
Bolt G, Austin J, Morgan G (1992) Fault tolerant multi-layer perceptron networks, Citeseer
Martolia R, Jain A, Singla L (2015) Analysis & survey on fault tolerance in radial basis function networks, In: International Conference on Computing, Communication & Automation, IEEE, pp 469–473
Murakami M, Honda N (2007) Fault tolerance comparison of ids models with multilayer perceptron and radial basis function networks, In: 2007 International Joint Conference on Neural Networks, IEEE, pp 1079–1084
Liu S-M, Tang L, Huang N-C, Tsai D-Y, Yang M-X, Wu K-C (2020) Fault-tolerance mechanism analysis on nvdla-based design using open neural network compiler and quantization calibrator, In: 2020 International symposium on VLSI design, automation and test (VLSI-DAT), IEEE, pp 1–3
Yamazaki K, Tsutsumi T, Takahashi H, Higami Y, Aikyo T, Takamatsu Y, Yotsuyanagi H, Hashizume M (2009) A novel approach for improving the quality of open fault diagnosis, In: 2009 22nd International Conference on VLSI Design, IEEE, pp 85–90
Leung CS, Wang H-J, Sum J (2010) On the selection of weight decay parameter for faulty networks. IEEE Trans Neural Netw 21(8):1232–1244
Leung C-S, Sum JP-F (2012) RBF networks under the concurrent fault situation. IEEE Trans Neural Netw Learn Syst 23(7):1148–1155
Feng R-B, Han Z-F, Wan WY, Leung C-S (2017) Properties and learning algorithms for faulty rbf networks with coexistence of weight and node failures. Neurocomputing 224:166–176
Konečnỳ J, Liu J, Richtárik P, Takáč M (2015) Mini-batch semi-stochastic gradient descent in the proximal setting. IEEE J Selected Top Signal Process 10(2):242–255
Qian Q, Jin R, Yi J, Zhang L, Zhu S (2015) Efficient distance metric learning by adaptive sampling and mini-batch stochastic gradient descent (sgd). Mach Learn 99(3):353–372
Amari S-I (1993) Backpropagation and stochastic gradient descent method. Neurocomputing 5(4–5):185–196
Li X, Orabona F (2019) On the convergence of stochastic gradient descent with adaptive stepsizes, In: The 22nd international conference on artificial intelligence and statistics, PMLR, pp 983–992
Bottou L et al (1991) Stochastic gradient learning in neural networks. Proc Neuro-Nımes 91(8):12
Cao Y, Gu Q (2019) Generalization bounds of stochastic gradient descent for wide and deep neural networks. Adv Neural Inform Process Syst 32:10836–10846
Cha E, Leung C-S, Wong E (2020) Convergence of mini-batch learning for fault aware rbf networks. In: Yang H, Pasupa K, Leung AC-S, Kwok JT, Chan JH, King I (eds) Neural Information Processing. Springer International Publishing, Cham, pp 545–553
Chen S (2006) Local regularization assisted orthogonal least squares regression. Neurocomputing 69(4–6):559–585
Asuncion A, Newman D (2007) Uci machine learning repository
Alcalá-Fdez J, Fernández A, Luengo J, Derrac J, García S, Sánchez L, Herrera F (2011) Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework., Journal of Multiple-Valued Logic & Soft Computing 17
Chang C-C, Lin C-J (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol (TIST) 2(3):1–27
Acknowledgements
This work is partially supported by Key Project of Science and Technology Innovation 2030 supported by the Ministry of Science and Technology of China (Grant No. 2018AAA0101301), GRF-RGC General Research Fund CityU 11203820 (9042958) and CityU 11209819 (9042816).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wong, H.T., Leung, CS. & Kwong, S. Convergence analysis on the deterministic mini-batch learning algorithm for noise resilient radial basis function networks. Int. J. Mach. Learn. & Cyber. 13, 2677–2690 (2022). https://doi.org/10.1007/s13042-022-01550-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-022-01550-6