Timeliness online regularized extreme learning machine

  • Xiong LuoEmail author
  • Xiaona Yang
  • Changwei Jiang
  • Xiaojuan Ban
Original Article


To improve the learning performance, a novel online sequential extreme learning machine (ELM) algorithm for single-hidden layer feedforward networks is proposed with regularization mechanism in a unified framework. The proposed algorithm is called timeliness online regularized extreme learning machine (TORELM). Like the timeliness managing extreme learning machine which improves online sequential extreme learning machine by incorporating timeliness management scheme into ELM approach for the incremental training samples, TORELM also analyzes the training data one-by-one or chunk-by-chunk (a block of data) with fixed or varied chunk size under the similar framework. Meanwhile, the newly incremental training data could be prior to the historical data by maximizing the contribution of the newly increasing training data, since in some cases it may be more feasible that the incremental data can contribute reasonable weights to represent the current system situation in accordance with the practical analysis. Furthermore, in consideration of the disproportion between empirical risk and structural risk in some traditional learning methods, we add regularization technique to the timeliness scheme of TORELM through the use of a weight factor to balance them to achieve better generalization performance. Hence, TORELM has its unique feature of higher generalization capability in most cases with a small testing error while implementing online sequential learning. In addition, this algorithm is still competitive in training time compared with other schemes. Finally, the simulation results show that TORELM can achieve higher learning accuracy and better stability than other ELM-based machine learning methods.


Incremental learning Extreme learning machine Timeliness online regularized extreme learning machine Regularization 



This research is funded by the National Natural Science Foundation of China under Grants 61174103, 61272357 and 61300074, the National Key Technologies R&D Program of China under Grant 2015BAK38B01, the Aerospace Science Foundation of China under Grant 2014ZA74001, and the Fundamental Research Funds for the Central Universities under Grant 06500025.


  1. 1.
    Huang GB, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122CrossRefGoogle Scholar
  2. 2.
    Luo X, Chang XH, Liu H (2014) A Taylor based localization algorithm for wireless sensor network using extreme learning machine. IEICE Trans Inf Syst E97D(10):2652–2659CrossRefGoogle Scholar
  3. 3.
    Luo X, Chang XH (2015) A novel data fusion scheme using grey model and extreme learning machine in wireless sensor networks. Int J Control Autom Syst 13(3):539–546CrossRefGoogle Scholar
  4. 4.
    You ZH, Lei YK, Zhu L, Xia JF, Wang B (2013) Prediction of protein-protein interactions from amino acid sequences with ensemble extreme learning machines and principal component analysis. BMC Bioinform 14(Suppl. 8):S10CrossRefGoogle Scholar
  5. 5.
    Huang GB, Bai Z, Kasun L, Vong C (2015) Local receptive fields based extreme learning machine. IEEE Comput Intell Magn 10(2):18–29CrossRefGoogle Scholar
  6. 6.
    Luo X, Chang XH, Ban XJ (2016) Regression and classification using extreme learning machine based on L1-norm and L2-norm. Neurocomputing. 174:179–186CrossRefGoogle Scholar
  7. 7.
    He YL, Wang XZ, Huang JZX (2016) Fuzzy nonlinear regression analysis using a random weight network. Inf Sci. doi: 10.1016/j.ins.2016.01.037 Google Scholar
  8. 8.
    Cambria E, Huang GB, Kasun LLC, Zhou H, Vong CM, Lin J, Yin J, Cai Z, Liu Q, Li K, Leung VCM, Feng L, Ong YS, Lim MH, Akusok A, Lendasse A, Corona F, Nian R, Miche Y, Gastaldo P, Zunino R, Decherchi S, Yang X, Mao K, Oh BS, Jeon J, Toh KA, Teoh ABJ, Kim J, Yu H, Chen Y, Liu J (2013) Extreme learning machines. IEEE Intell Syst 28(6):30–59CrossRefGoogle Scholar
  9. 9.
    Wang XZ, Chen AX, Feng HM (2011) Upper integral network with extreme learning mechanism. Neurocomputing 74(16):2520–2525CrossRefGoogle Scholar
  10. 10.
    Lu SX, Wang XZ, Zhang GQ, Zhou X (2015) Effective algorithms of the Moore–Penrose inverse matrices for extreme learning machine. Intell Data Anal 19(4):743–760CrossRefGoogle Scholar
  11. 11.
    Chacko BP, Krishnan VRV, Raju G (2012) Handwritten character recognition using wavelet energy and extreme learning machine. Int J Mach Learn Cybern 3(2):149–161CrossRefGoogle Scholar
  12. 12.
    Fu A, Dong C, Wang L (2015) An experimental study on stability and generalization of extreme learning machines. Int J Mach Learn Cybern 6(1):129–135CrossRefGoogle Scholar
  13. 13.
    Viskan K, Mahesan N (1993) A function estimation approach to sequential learning with neural networks. Neural Comput 5(6):954–975CrossRefGoogle Scholar
  14. 14.
    Nobrega JP, Oliveira ALI (2015) Kalman filter-based method for online sequential extreme learning machine for regression problems. Eng Appl Artif Intell 44:101–110CrossRefGoogle Scholar
  15. 15.
    Kadirkamanathan V, Niranjan M (1993) A function estimation approach to sequential learning with neural networks. Neural Comput 5(6):954–975CrossRefGoogle Scholar
  16. 16.
    Liang N, Huang GB, Saratchandran P, Sundararajan N (2006) A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Trans Neural Netw 17(6):1411–1423CrossRefGoogle Scholar
  17. 17.
    Scardapane S, Comminiello D, Scarpiniti M (2015) Online sequential extreme learning machine with kernels. IEEE Trans Neural Netw Learn Syst 26(9):2214–2220MathSciNetCrossRefGoogle Scholar
  18. 18.
    Matias T, Souza F, Araújo R, Gonçalves N, Barreto JP (2015) On-line sequential extreme learning machine based on recursive partial least squares. J Process Control 27:15–21CrossRefGoogle Scholar
  19. 19.
    Yuan L, Soh YC, Huang GB (2009) Ensemble of online sequential extreme learning machine. Neurocomputing 72(13):3391–3395Google Scholar
  20. 20.
    Liu JF, Gu Y, Chen YQ, Cao YS (2013) Incremental localization in WLAN environment with timeliness management. Chin J Comput 36(7):1448–1455CrossRefGoogle Scholar
  21. 21.
    Vapnik VN (1995) The nature of statistical learning theory. IEEE Trans Neural Netw 10(5):988–999MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Deng WY, Zheng QH, Chen L (2009) Regularized extreme learning machine. In: Proceedings of the IEEE symposium on computational intelligence and data mining, pp 389–395Google Scholar
  23. 23.
    Huang GB, Ding X, Zhou H (2010) Optimization method based extreme learning machine for classification. Neurocomputing 74(1–3):155–163CrossRefGoogle Scholar
  24. 24.
    Huang GB, Zhou H, Ding X (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B Cybern 42(2):513–529CrossRefGoogle Scholar
  25. 25.
    Gu Y, Liu JF, Chen YQ, Jiang XL, Yu HC (2014) TOSELM: timeliness online sequential extreme learning machine. Neurocomputing 128:119–127CrossRefGoogle Scholar
  26. 26.
    Glenn F, Mangasarian OL (2001) Proximal support vector machine classifiers. In: Proceedings of the 7th ACM SIGKDD international conference on knowledge discovery and data mining, pp 77–86Google Scholar
  27. 27.
    David HA (1998) Early sample measures of variability. Stat Sci 13(4):368–377MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Rousseeuw PJ, Leroy A (1987) Robut regression and outlier detection. Wiley, New YorkCrossRefzbMATHGoogle Scholar
  29. 29.
    Miche Y, Sorjamaa A, Bas P, Simula O, Jutten C, Lendasse A (2010) OP-ELM: optimally pruned extreme learning machine. IEEE Trans Neural Netw 21(1):158–162CrossRefGoogle Scholar
  30. 30.
    Wang XZ, Shao QY, Qing M, Zhai JH (2013) Architecture selection for networks trained with extreme learning machine using localized generalization error model. Neurocomputing 102:3–9CrossRefGoogle Scholar
  31. 31.
    Stock Data (2016) YAHOO Finance API. Accessed 5 May 2016
  32. 32.
    Graham L, Oppacher F (2008) UCI machine learning repository, University of California, Irvine, School of Information and Computer Sciences. Accessed 5 May 2016
  33. 33.
    Dias DB, Peres SM, Bíscaro HH (2009) UCI machine learning repository, University of California, Irvine, School of Information and Computer Sciences. Accessed 5 May 2016
  34. 34.
    Roesler O (2013) UCI machine learning repository, University of California, Irvine, School of Information and Computer Sciences. Accessed 5 May 2016

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Xiong Luo
    • 1
    • 2
    Email author
  • Xiaona Yang
    • 1
    • 2
  • Changwei Jiang
    • 1
    • 2
  • Xiaojuan Ban
    • 1
    • 2
  1. 1.School of Computer and Communication EngineeringUniversity of Science and Technology BeijingBeijingPeople’s Republic of China
  2. 2.Beijing Key Laboratory of Knowledge Engineering for Materials ScienceBeijingPeople’s Republic of China

Personalised recommendations