Abstract
Extreme learning machine (ELM) is a non-iterative algorithm for training single-hidden layer feed-forward networks, whose training speed is much faster than those of conventional neural networks. However, its objective is only to minimize the empirical risk, which may cause overfitting easily. To overcome this defect, this paper proposes a novel algorithm named LGE2LM based on the localized generalization error model which provides an upper bound of generalization error through adopting the stochastic sensitivity. LGE2LM aims to improve the generalization capability of ELM by using the regularization technique which makes a trade-off between the empirical risk and the stochastic sensitive measure. The essence of LGE2LM is a quadratic problem without iterative process. Similar to ELM, all the parameters of this new algorithm are obtained without tuning, which makes the efficiency of LGE2LM much higher than those of traditional neural networks. Several experiments conducted on both artificial and real-world datasets show that LGE2LM has much better generalization capability and stronger robustness in comparison with ELM.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Asuncion A, Newman D (2007) UCI machine learning repository. http://www.ics.uci.edu/mlearn/MLRepository
Bangalore P, Tjernberg LB (2017) An artificial neural network approach for early fault detection of gearbox bearings. IEEE Trans Smart Grid 6(2):980–987. https://doi.org/10.1109/TSG.2014.2386305
Bouhoune K, Yazid K, Boucherit MS et al (2017) Hybrid control of the three phase induction machine using artificial neural networks and fuzzy logic. Appl Soft Comput 55(C):289–301
Dash Y, Mishra S K, Sahany S, et al. (2017) Indian summer monsoon rainfall prediction: a comparison of iterative and non-iterative approaches. Appl Soft Comput https://doi.org/10.1016/j.asoc.2017.08.055
Ding S, Zhang N, Zhang J et al (2017) Unsupervised extreme learning machine with representational features. Int J Mach Learn Cybern 8(2):587–595
Fontenla-Romero O, Pérez-Sánchez B, Guijarro-Berdiñas B (2017) LANN-SVD: a non-iterative SVD-based learning algorithm for one-layer neural networks. IEEE Trans Neural Netw Learn Syst 99:1–6
Gheisari S, Meybodi MR, Dehghan M, Ebadzadeh MM (2017) Bayesian network structure training based on a game of learning automata. Int J Mach Learn Cybern 8(4):1093–1105. https://doi.org/10.1007/s13042-015-0476-9
Ham FM, Kostanic I (2003) Principles of neurocomputing for science and engineering. McGraw-Hill, New York
Haykin S (2004) Neural networks: a comprehensive foundation, 2nd edn. Prentice Hall, Upper Saddle River
Hoeffding W (1963) Probability inequalities for sums of bounded random variables. J Am Stat Assoc 58(1):13–30. https://doi.org/10.1007/978-1-4612-0865-5_26
Hua M, Tan H, Fei J (2017) State estimation for uncertain discrete-time stochastic neural networks with Markovian jump parameters and time-varying delays. Int J Mach Learn Cybern 8(3):823–835. https://doi.org/10.1007/s13042-015-0373-2
Huang Z (2017) Almost periodic solutions for fuzzy cellular neural networks with multi-proportional delays. Int J Mach Learn Cybern 8(4):1323–1331. https://doi.org/10.1007/s13042-016-0507-1
Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501. https://doi.org/10.1016/j.neucom.2005.12.126
Huang Z, Yu Y, Gu J et al (2017) An efficient method for traffic sign recognition based on extreme learning machine. IEEE Trans Cybern 99(1):1–14. https://doi.org/10.1109/TCYB.2016.2533424
Liu Y (1995) Unbiased estimate of generalization error and model selection in neural network. Neural Netw 8(2):215–219. https://doi.org/10.1016/0893-6080(94)00089-5
Liu T, Tao D, Song M et al (2017) Algorithm-dependent generalization bounds for multi-task learning. IEEE Trans Pattern Anal Mach Intell 39(2):227–241. https://doi.org/10.1109/TPAMI.2016.2544314
Mao W, Wang J, Xue Z (2017) An ELM-based model with sparse-weighting strategy for sequential data imbalance problem. Int J Mach Learn Cybern 8(4):1333–1345. https://doi.org/10.1007/s13042-016-0509-z
Mitchell T (1996) Machine learning. McGraw-Hill, New York
Ng WWY, Yeung DS, Firth M et al (2008) Feature selection using localized generalization error for supervised classification problems using RBFNN. Pattern Recognit 41(12):3706–3719. https://doi.org/10.1016/j.patcog.2008.05.004
Polhill JG, Weir MK (2001) An approach to guaranteeing generalization in neural networks. Neural Netw 14(8):1035–1048. https://doi.org/10.1016/S0893-6080(01)00061-2
Sadjina S, Kyllingstad LT, Skjong S et al (2017) Energy conservation and power bonds in co-simulations: non-iterative adaptive step size control and error estimation. Eng Comput 2017:1–14
Sheng C, Zhao J, Wang W (2017) Map-reduce framework-based non-iterative granular echo state network for prediction intervals construction. Neurocomputing 222:116–126
Sun BB, Ng WWY, Chan PPK (2017) Improved sparse LSSVMS based on the localized generalization error model. Int J Mach Learn Cybern 8(6):1853–1861. https://doi.org/10.1007/s13042-016-0563-6
Wang Q, Chen G (2017) Fuzzy soft subspace clustering method for gene co-expression network analysis. Int J Mach Learn Cybern 8(4):1157–1165. https://doi.org/10.1007/s13042-015-0486-7
Wang XZ, Dong CR (2009) Improving generalization of fuzzy if–then rules by maximizing fuzzy entropy. IEEE Trans Fuzzy Syst 17(3):556–567. https://doi.org/10.1109/TFUZZ.2008.924342
Wang XZ, Zhai JH, Lu SX (2008) Induction of multiple fuzzy decision trees based on rough set technique. Inf Sci 178(16):3188–3202. https://doi.org/10.1016/j.ins.2008.03.021
Wang XZ, Shao QY, Miao Q et al (2013) Architecture selection for networks trained with extreme learning machine using localized generalization error model. Neurocomputing 102:3–9. https://doi.org/10.1016/j.neucom.2011.12.053
Wang N, Meng JE, Han M (2017) Parsimonious extreme learning machine using recursive orthogonal least squares. IEEE Trans Neural Netw Learn Syst 25(10):1828–1841. https://doi.org/10.1109/TNNLS.2013.2296048
Wang C, Yuan J, Xie L (2017) Non-iterative SLAM. In: International conference on advanced robotics. IEEE 2017, pp 83–90
Xiao J, Zhong S, Li Y et al (2017) Relaxed exponential passivity criteria for memristor-based neural networks with leakage and time-varying delays. Int J Mach Learn Cybern 8(6):1875–1886. https://doi.org/10.1007/s1304
Yan W, Hou S, Fang Y, Fei J (2017) Robust adaptive nonsingular terminal sliding mode control of MEMS gyroscope using fuzzy-neural-network compensator. Int J Mach Learn Cybern 8(4):1287–1299. https://doi.org/10.1007/s13042-016-0501-7
Yeung DS, Ng WWY, Wang DF et al (2007) Localized generalization error and its application to architecture selection for radial basis function neural network. IEEE Trans Neural Netw 18(5):1294–1305. https://doi.org/10.1109/TNN.2007.894058
Yeung DS, Chan PPK, Ng WWY (2009) Radial basis function network learning using localized generalization error bound. Inf Sci 179(19):3199–3217. https://doi.org/10.1016/j.ins.2009.06.001
Zhang Y (2017) A projected-based neural network method for second-order cone programming. Int J Mach Learn Cybern 8(6):1907–1914. https://doi.org/10.1007/s1304
Acknowledgements
This work is supported by the Macao Science and Technology Development Funds (100/2013/A3 and 081/2015/A3), the National Natural Science Foundation of China (Grant 61772344 and Grant 61732011), in part by the Natural Science Foundation of SZU (Grant 827-000140, Grant 827-000230, and Grant 2017060), the National Social Science Foundation of China (17BTQ068), the Youth Foundation Project of Hebei Natural Science Foundation of China (F2018511002).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Communicated by X. Wang, M. Pelillo, A. K. Sangaiah.
Rights and permissions
About this article
Cite this article
Zhu, H., Tsang, E.C.C. & Zhu, J. Training an extreme learning machine by localized generalization error model. Soft Comput 22, 3477–3485 (2018). https://doi.org/10.1007/s00500-018-3012-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-018-3012-5