Artificial Intelligence Review

, Volume 44, Issue 1, pp 103–115 | Cite as

Extreme learning machine: algorithm, theory and applications

  • Shifei Ding
  • Han Zhao
  • Yanan Zhang
  • Xinzheng Xu
  • Ru Nie
Article

Abstract

Extreme learning machine (ELM) is a new learning algorithm for the single hidden layer feedforward neural networks. Compared with the conventional neural network learning algorithm it overcomes the slow training speed and over-fitting problems. ELM is based on empirical risk minimization theory and its learning process needs only a single iteration. The algorithm avoids multiple iterations and local minimization. It has been used in various fields and applications because of better generalization ability, robustness, and controllability and fast learning rate. In this paper, we make a review of ELM latest research progress about the algorithms, theory and applications. It first analyzes the theory and the algorithm ideas of ELM, then tracking describes the latest progress of ELM in recent years, including the model and specific applications of ELM, finally points out the research and development prospects of ELM in the future.

Keywords

Extreme learning machine (ELM) Single-hidden layer feedforward neural networks (SLFNs) Local minimum Over-fitting Least-squares 

References

  1. Benqio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127CrossRefGoogle Scholar
  2. Cai L, Cheng G, Pan H (2010) Lithologic identification based on ELM. Comput Engi Des 31(9):210–2012Google Scholar
  3. Carpenter G, Grossberg S (2003) Adaptive resonance theory. In: Arbib MA (ed) The handbook of brain theory and neural networks, 2nd edn. MIT Press, Cambridge, pp 87–90Google Scholar
  4. Chang Y, Li Y, Wang F (2007) Soft sensing modeling based on extreme learning machine for biochemical processes. J Syst Simul 19(23):5587–5590Google Scholar
  5. Deng W, Zheng Q, Chen L et al (2010a) Research on extreme learning of neural networks. Chin J Comput 33(2):279–287CrossRefMathSciNetGoogle Scholar
  6. Deng W, Zheng Q, Lian S et al (2010b) Ordinal extreme learning machine. Neurocomputing 74(1–3):447–456CrossRefGoogle Scholar
  7. Ding S, Jia W, Su C et al (2011a) Research of neural network algorithm based on factor analysis and cluster analysis. Neural Comput Appl 20(2):297–302CrossRefGoogle Scholar
  8. Ding S, Su C, Yu J et al (2011b) An optimizing BP neural network algorithm based on genetic algorithm. Artif Intell Rev 36(2):153–162CrossRefGoogle Scholar
  9. Ding S, Xu L, Chunyang SuC et al (2012) An optimizing method of RBF neural network based on genetic algorithm. Neural Comput Appl 21(2):333–336CrossRefGoogle Scholar
  10. Ding S, Jin F, Zhao X (2013) Modern data analysis and information pattern recognition. Science Press, BeijingGoogle Scholar
  11. Feng G, Huang G, Lin Q et al (2009) Error minimized extreme learning machine with growth of hidden nodes and incremental learning. IEEE Trans Neural Netw 20(8):1352–1357CrossRefGoogle Scholar
  12. Fernandez-Navarro F, Hervas-Martinez C, Sanchez-Monedero J et al (2011) MELM-GRBF: A modified version of the extreme learning machine for generalized radial basis function neural networks. Neurocomputing 74(16):2502–2510CrossRefGoogle Scholar
  13. Hagan MT, Demuth HB, Beale MH (2002) Mechanical Industry Press, Beijing, ChinaGoogle Scholar
  14. Han F, Huang D (2006) Improved extreme learning machine for function approximation by encoding apriori information. Neurocomputing 69(16–18):2369–2373CrossRefGoogle Scholar
  15. Hornik K (1991) Approximation capabilities of multilayer feedforward networks. Neural Netw 4(2):251–257CrossRefGoogle Scholar
  16. Huang G (2003) Learning capability and storage capacity of two hidden-layer feedforward networks. IEEE Trans Neural Netw 14(2):274–281CrossRefGoogle Scholar
  17. Huang G (2003) Learning capability and storage capacity of two-hidden-layer feedforward network. IEEE Trans Neural Netw 14(2):274–281CrossRefGoogle Scholar
  18. Huang G, Babri HA (1998) Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans Neural Netw 9(1):224–229CrossRefGoogle Scholar
  19. Huang G, Liang N, Rong HJ et al. (2005) On-line sequential extreme learning machine. In: The IASTED international conference on, computational intelligence, pp. 232–237Google Scholar
  20. Huang G, Wang D, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122CrossRefGoogle Scholar
  21. Huang G, Zhu Q, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. IEEE Int Jt Conf Neural Netw 1–4:985–990Google Scholar
  22. Huang G, Zhu Q, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501CrossRefGoogle Scholar
  23. Kahramanli H, Allahverdi N (2009) Rule extraction from trained adaptive neural networks using artificial immune systems. Expert Syst Appl 36(2):1513–1522CrossRefGoogle Scholar
  24. Lan Y, Soh YC, Huang G (2010) Two-stage extreme learning machine for regression. Neurocomputing 73(2010):3028–3038CrossRefGoogle Scholar
  25. LeCun Y, Bengio Y (1995) Convolutional networks for images, speech, and time series. MIT Press, CambridgeGoogle Scholar
  26. Leshno M, Lin V, Pinkus A et al (1991) Multilayer feedforward networks with a non-polynomial activation function can approximate any function. Neural Netw 6(6):861–867CrossRefGoogle Scholar
  27. Li S, Chen SF, Liu B (2013) Accelerating a recurrent neural network to finite-time convergence for solving time-varying Sylvester equation by using a sign-bi-power activation function. Neural Process Lett 37:189–205CrossRefGoogle Scholar
  28. Li S, Chen SF, Liu B et al (2012) Decentralized kinematic control of a class of collaborative redundant manipulators via recurrent neural networks. Neurocomputing 91:1–10CrossRefMathSciNetGoogle Scholar
  29. Li S, Liu B, Li YM (2013) Selective positive-negative feedback produces the winner-take-all competition in recurrent neural networks. IEEE Trans Neural Netw Learn Syst 24(2):301–309CrossRefGoogle Scholar
  30. Li S, Wang Y, Yu J et al (2013) A nonlinear model to generate the winner-take-all competition. Commun Nonlinear Sci Numer Simul 18(3):435–442CrossRefMATHMathSciNetGoogle Scholar
  31. Liang N, Huang G (2006) A fast and accurate online sequential learning algorithm for feed forward networks. IEEE Trans Neural Netw 17(6):1411–1423CrossRefGoogle Scholar
  32. Mao K (2002) RBF neural network center selection based on Fisher ratio class separability measure. IEEE Trans Neural Netw 13(5):1211–1217CrossRefGoogle Scholar
  33. Mao K, Huang G (2005) Neuron selection for RBF neural network classifier based on data structure preserving criterion. IEEE Trans Neural Netw 16(6):1531–1540CrossRefGoogle Scholar
  34. Markowska-Kaczmar U, Trelak W (2005) Fuzzy logic and evolutionary algorithm-two techniques in rule extraction from neural networks. Neurocomputing 63:359–379CrossRefGoogle Scholar
  35. Martinez-Martinez JM, Escandell-Montero P, Soria-Olivas E et al (2011) Regularized extreme learning machine for regression problems. Neurocomputing 74(17):3716–3721CrossRefGoogle Scholar
  36. McCulloch W, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5(4):115–133Google Scholar
  37. Mohamed MH (2011) Genetic algorithms. Neurocomputing 74(17):3180–3192CrossRefGoogle Scholar
  38. Mohammed AA, Minhas R, Wu Q et al (2011) Human face recognition based on multidimensional PCA and extreme learning machine. Pattern Recognit 44(10–11):2588–2597CrossRefMATHGoogle Scholar
  39. Pan H, Cheng G, Cai L (2010) Comparison of the extreme learning machine with the support vector machine for reservoir permeability prediction. Comput Engi Sci 32(2):131–134Google Scholar
  40. Quteishat A, Lim CP (2008) A modified fuzzy min-max neural network with rule extraction and its application to fault detection and classification. Appl Soft Comput 8(2):985–995CrossRefGoogle Scholar
  41. Romero E, Alquezar R (2012) Comparing error minimized extreme learning machines and support vector sequential feed-forward neural networks. Neural Netw 25:122–129CrossRefGoogle Scholar
  42. Rong HJ, Huang G, Sundararajan N et al (2009) Online sequential fuzzy extreme learning machine for function approximation and classification problems. IEEE Trans Syst Man Cybern Part B Cybern 39(4):1067–1072CrossRefGoogle Scholar
  43. Rong HJ, Ong YS, Tan AH et al (2008) A fast pruned-extreme learning machine for classification problem. Neurocomputing 72(1–3):359–366CrossRefGoogle Scholar
  44. Shang L, Wang J, Yao W (2005) A classification approach based on evolutionary neural networks. J Softw 16(9):1577–1583CrossRefMATHMathSciNetGoogle Scholar
  45. Silva DNG, Pacifico LDS, Ludermir TB (2011) An evolutionary extreme learning machine based on group search optimization. IEEE Congress on Evolutionary Computation, pp. 574–580Google Scholar
  46. Wang Y, Cao F, Yuan Y (2011) A study on effectiveness of extreme learning machine. Neurocomputing 74(16):2483–2490CrossRefGoogle Scholar
  47. Xu X, Ding S, Shi Z et al (2012) A novel optimizing method for RBF neural network based on rough set and AP clustering algorithm. J Zhejiang University-SCIENCE C 13(2):131–138CrossRefGoogle Scholar
  48. Yao W, Han M (2010) Fusion of thermal infrared and multispectral remote sensing images via neural network regression. J Image Graphics 15(8):1278–1284Google Scholar
  49. Zhang D, Wang Y (2009) Rough neural network based on bottom-up fuzzy rough data analysis. Neural Process Lett 30(3):187–211CrossRefGoogle Scholar
  50. Zhang X, Wang H (2011) Incremental regularized extreme learning machine based on Cholesky factorization and its application to time series prediction. Acta Physica Sinica 60(11):110201-1–111201-6Google Scholar
  51. Zhang X, Wang H (2011) Selective forgetting extreme learning machine and its application to time series prediction. Acta Physica Sinica, 60(8):080504-1–080504-6Google Scholar
  52. Zhu Q, Qin A, Suganthan PN et al (2005) Evolutionary extreme learning machine. Patt Recognit 38(10):1759–1763CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Shifei Ding
    • 1
    • 2
  • Han Zhao
    • 1
  • Yanan Zhang
    • 1
  • Xinzheng Xu
    • 1
  • Ru Nie
    • 1
  1. 1.School of Computer Science and TechnologyChina University of Mining and TechnologyXuzhouChina
  2. 2.Key Laboratory of Intelligent Information Processing, Institute of Computing TechnologyChinese Academy of ScienceBeijingChina

Personalised recommendations