Skip to main content
Log in

Metaheuristic-based extreme learning machines: a review of design formulations and applications

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Extreme learning machine (ELM) is a novel and recent machine learning algorithm which was first proposed by Huang et al. (Proceedings of 2004 IEEE international joint conference on, pp 985–990, 2004). Over the last decade, ELM has gained a remarkable research interest with tremendous audiences from different domains in a short period of time due to its impressive characteristics over other single hidden-layer feedforward neural networks. Although ELM enjoys powerful advantages, it still has some potential weaknesses like performance sensitivity to the initial condition of the input weights, number of hidden neurons, and the selection of activation functions. In order to overcome the limitations of classical ELM, many metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, memetic and trajectory algorithms have been proposed for optimizing the different components of ELM by researchers aiming to improve the generalization performance of ELM networks for different types of complex problems and applications. Therefore our review paper intent to conduct a deep study of the important aspects of applying metaheuristic algorithms for optimizing ELM networks. Three main streams of research lines are identified: the optimization of input weights and hidden biases, selection of hidden neurons, and optimization of activation functions. Furthermore, this paper will discuss a wide spectrum of applications of metaheuristic-based ELM models. We will highlight the strengths of these models and the improvements that are suggested in the literature to overcome their weaknesses. We touch upon several interesting and challenging open issues in optimizing ELM using metaheuristics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Alade OA , Selamat A, Sallehuddin R (2017) A review of advances in extreme learning machine techniques and its applications. In: International conference of reliable information and communication technology. Springer, pp 885–895

  2. Alexandridis A, Famelis IT, Tsitouras C, Simos T, Tsitouras C (2016) Particle swarm optimization for complex nonlinear optimization problems. In: AIP conference proceedings, vol 1738. AIP Publishing, pp 480120

  3. Aljarah I, Ludwig SA (2012) Parallel particle swarm optimization clustering algorithm based on mapreduce methodology. In: Nature and biologically inspired computing (NaBIC), 2012 fourth world congress on. IEEE, pp 104–111

  4. Aljarah I, Ludwig SA (2013) A new clustering approach based on glowworm swarm optimization. In: Evolutionary computation (CEC), 2013 IEEE congress on. IEEE, pp 2642–2649

  5. Almeida LB (1989) Backpropagation in perceptrons with feedback. In: Neural computers. Springer, pp199–208

  6. Arunadevi B, Deepa SN (2013) Brain tumor tissue categorization in 3D magnetic resonance images using improved PSO for extreme learning machine. Progress Electromagn Res B 49:31–54

    Article  Google Scholar 

  7. Azzini A, Tettamanzi AGB (2011) Evolutionary anns: a state of the art survey. Intell Artif 5(1):19–35

    Google Scholar 

  8. Bartlett PL (1998) The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Trans Inf Theory 44(2):525–536

    Article  MathSciNet  MATH  Google Scholar 

  9. Bi X, Zhao X, Wang G, Zhang P, Wang C (2015) Distributed extreme learning machine with kernels based on mapreduce. Neurocomputing 149:456–463

    Article  Google Scholar 

  10. Leonora B, Marco D, Gambardella LM, Gutjahr WJ (2009) A survey on metaheuristics for stochastic combinatorial optimization. Nat Comput 8(2):239–287

    Article  MathSciNet  MATH  Google Scholar 

  11. Blum C, Roli A (2003) Metaheuristics in combinatorial optimization: overview and conceptual comparison. ACM Computi Surv 35(3):268–308

    Article  Google Scholar 

  12. BoussaïD I, Lepagnot J, Siarry P (2013) A survey on optimization metaheuristics. Inf Sci 237:82–117

    Article  MathSciNet  MATH  Google Scholar 

  13. Cao J, Lin Z (2015) Extreme learning machines on high dimensional and large data applications: a survey. Math Probl Eng

  14. Cao J, Lin Z, Huang G-B (2012) Self-adaptive evolutionary extreme learning machine. Neural Process Lett 36(3):285–305

    Article  Google Scholar 

  15. Cho J-H, Lee D-J, Chun M-G (2007) Parameter optimization of extreme learning machine using bacterial foraging algorithm. J Korean Ins Intell Syst 17(6):807–812

    Article  Google Scholar 

  16. Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Math Control Signals Syst 2(4):303–314

    Article  MathSciNet  MATH  Google Scholar 

  17. de Oliveira JFL, Ludermir TB (2012) An evolutionary extreme learning machine based on fuzzy fish swarms. In: Proceedings on the international conference on artificial intelligence (ICAI), p 1. The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp)

  18. Ding S, Chunyang S, Junzhao Y (2011) An optimizing bp neural network algorithm based on genetic algorithm. Artif Intell Rev 36(2):153–162

    Article  Google Scholar 

  19. Ding S, Xinzheng X, Nie R (2014) Extreme learning machine and its applications. Neural Comput Appl 25(3–4):549–556

    Article  Google Scholar 

  20. Ding S, Zhao H, Zhang Y, Xinzheng X, Nie R (2015) Extreme learning machine: algorithm, theory and applications. Artif Intell Rev 44(1):103–115

    Article  Google Scholar 

  21. Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern Part B Cybern 26(1):29–41

    Article  Google Scholar 

  22. Eberhart R, Kennedy J (1995) A new optimizer using particle swarm theory. In: Micro machine and human science, 1995. MHS’95., Proceedings of the sixth international symposium on. IEEE, pp 39–43

  23. Ertuğrul ÖF, Kaya Y (2014) A detailed analysis on extreme learning machine and novel approaches based on elm. Am J Comput Sci Eng 1(5):43–50

    Google Scholar 

  24. Eshtay M, Faris H, Obeid N (2018) Improving extreme learning machine by competitive swarm optimization and its application for medical diagnosis problems. Expert Syst Appl 104:134

    Article  Google Scholar 

  25. Faris H, Aljarah I, Mirjalili S (2016) Training feedforward neural networks using multi-verse optimizer for binary classification problems. App Intell 45(2):322–332

    Article  Google Scholar 

  26. Faris H, Aljarah I, Al-Madi N, Mirjalili S (2016) Optimizing the Learning Process of Feedforward Neural Networks Using Lightning Search Algorithm. Int J Artif Intell Tools 25(06):1650033

    Article  Google Scholar 

  27. Feo TA, Resende MGC (1995) Greedy randomized adaptive search procedures. J Glob Optim 6(2):109–133

    Article  MathSciNet  MATH  Google Scholar 

  28. Freire A, Barreto G (2014) A new model selection approach for the elm network using metaheuristic optimization. In: European symposium on artificial neural networks, computational intelligence and machine learning (ESANN)

  29. Glover F (1989) Tabu search part I. ORSA J Comput 1(3):190–206

    Article  MathSciNet  MATH  Google Scholar 

  30. Gori M, Tesi A (1992) On the problem of local minima in backpropagation. IEEE Trans Pattern Anal Mach Intell 1:76–86

    Article  Google Scholar 

  31. Gupta JND, Sexton RS (1999) Comparing backpropagation with a genetic algorithm for neural network training. Omega 27(6):679–684

    Article  Google Scholar 

  32. Hagan MT, Menhaj MB (1994) Training feedforward networks with the Marquardt algorithm. IEEE Trans Neural Netw 5(6):989–993

    Article  Google Scholar 

  33. Han F, Yao H-F, Ling Q-H (2013) An improved evolutionary extreme learning machine based on particle swarm optimization. Neurocomputing 116:87–93

    Article  Google Scholar 

  34. Hart WE, Krasnogor N, Smith JE (2004) Recent advances in memetic algorithms, volume 166. Springer, Berlin

    Google Scholar 

  35. Ho Y-C, Pepyne DL (2002) Simple explanation of the no-free-lunch theorem and its implications. J Optim Theory Appl 115(3):549–570

    Article  MathSciNet  MATH  Google Scholar 

  36. Hoerl AE, Kennard RW (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1):55–67

    Article  MATH  Google Scholar 

  37. Holland JH (1992) Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT Press, Cambridge

    Book  Google Scholar 

  38. Hopfield JJ (1982) Neural networks and physical systems with emergent collective computational abilities. Proc Nat Acad Sci 79(8):2554–2558

    Article  MathSciNet  MATH  Google Scholar 

  39. Hornik K (1991) Approximation capabilities of multilayer feedforward networks. Neural Netw 4(2):251–257

    Article  MathSciNet  Google Scholar 

  40. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366

    Article  MATH  Google Scholar 

  41. Huang G, Huang G-B, Song S, You K (2015) Trends in extreme learning machines: a review. Neural Netw 61:32–48

    Article  MATH  Google Scholar 

  42. Huang G-B (2015) What are extreme learning machines? Filling the gap between frank Rosenblatts dream and John von Neumanns puzzle. Cogn Comput 7(3):263–278

    Article  Google Scholar 

  43. Huang G-B, Babri HA (1998) Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans Neural Netw 9(1):224–229

    Article  Google Scholar 

  44. Huang G-B, Bai Z, Kasun LLC, Vong CM (2015) Local receptive fields based extreme learning machine. IEEE Comput Intell Mag 10(2):18–29

    Article  Google Scholar 

  45. Huang G-B, Chen L, Siew CK et al (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892

    Article  Google Scholar 

  46. Guang-Bin H, Wang DH, Lan Y (2011) Extreme learning machines: a survey. Int J Mach Learn Cybern 2(2):107–122

    Article  Google Scholar 

  47. Huang G-B, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B Cybern 42(2):513–529

    Article  Google Scholar 

  48. Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: Neural networks, 2004. Proceedings of 2004 IEEE international joint conference on, vol 2. IEEE, pp 985–990

  49. Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501

    Article  Google Scholar 

  50. Huang Q, Jiang C, Huang Y (2016) The prediction method of SO\(\_2\) concentration in sulfuric acid production process based on GA-ELM. In: Intelligent human-machine systems and cybernetics (IHMSC), 2016 8th international conference on, vol 2. IEEE, pp 140–143

  51. Huang Y, Lai D (2012) Hidden node optimization for extreme learning machine. Aasri Procedia 3:375–380

    Article  Google Scholar 

  52. Huynh HT, Won Y (2008) Evolutionary algorithm for training compact single hidden layer feedforward neural networks. In: International joint conference on neural networks (IJCNN). IEEE, pp 3028–3033

  53. Karaboga D (2005) An idea based on honey bee swarm for numerical optimization. Technical report, Technical report-tr06, Erciyes University, Engineering Faculty, Computer Engineering Department

  54. Karlik B, Olgac AV (2011) Performance analysis of various activation functions in generalized MLP architectures of neural networks. Int J Artif Intell Expert Syst 1(4):111–122

    Google Scholar 

  55. Kasun LLC, Zhou H, Huang G-B, Vong CM (2013) Representational learning with elms for big data

  56. Kaya Y, Kayci L, Tekin R, Ertuğrul ÖF (2014) Evaluation of texture features for automatic detecting butterfly species using extreme learning machine. J Exp Theor Artif Intell 26(2):267–281

    Article  Google Scholar 

  57. Kirkpatrick S, Gelatt CD, Vecchi MP et al (1983) Optimization by simulated annealing. Science 220(4598):671–680

    Article  MathSciNet  MATH  Google Scholar 

  58. Kohonen T (1990) The self-organizing map. Proc IEEE 78(9):1464–1480

    Article  Google Scholar 

  59. Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection, vol 1. MIT Press, Cambridge

    MATH  Google Scholar 

  60. Krasnogor N, Smith J (2005) A tutorial for competent memetic algorithms: model, taxonomy, and design issues. IEEE Trans Evol Comput 9(5):474–488

    Article  Google Scholar 

  61. Leshno M, Lin VY, Pinkus A, Schocken S (1993) Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw 6(6):861–867

    Article  Google Scholar 

  62. Li B, Li Y, Rong X (2013) The extreme learning machine learning algorithm with tunable activation function. Neural Comput Appl 22(3–4):531–539

    Article  Google Scholar 

  63. Li S, Wang P, Goel L (2015) Short-term load forecasting by wavelet transform and evolutionary extreme learning machine. Electr Power Syst Res 122:96–103

    Article  Google Scholar 

  64. Li X, Shao Z, Qian J et al (2002) An optimizing method based on autonomous animats: fish-swarm algorithm. Syst Eng Theory Pract 22(11):32–38

    Google Scholar 

  65. Lin M, Zhang C, Su C (2016) Prediction of effluent from WWTPS using differential evolutionary extreme learning machines. In: Control conference (CCC), 2016 35th Chinese. IEEE, pp 2034–2038

  66. Liu N, Wang H (2013) Evolutionary extreme learning machine and its application to image analysis. J Signal Proces Syst 73(1):73–81

    Article  Google Scholar 

  67. Malhotra R, Singh N, Singh Y (2011) Genetic algorithms: concepts, design for optimization of process controllers. Comput Inf Sci 4(2):39

    Google Scholar 

  68. Matias T, Araújo R, Antunes CH, Gabriel D (2013) Genetically optimized extreme learning machine. In: Emerging technologies and factory automation (ETFA), 2013 IEEE 18th conference on. IEEE, pp 1–8

  69. Matias T, Souza F, Arajo R, Antunes CH (2014) Learning of a single-hidden layer feedforward neural network using an optimized extreme learning machine. Neurocomputing 129:428–436

    Article  Google Scholar 

  70. Mladenović N, Hansen P (1997) Variable neighborhood search. Comput Oper Res 24(11):1097–1100

    Article  MathSciNet  MATH  Google Scholar 

  71. Mohapatra P, Chakravarty S, Dash PK (2015) An improved cuckoo search based extreme learning machine for medical data classification. Swarm Evol Comput 24:25–49

    Article  Google Scholar 

  72. Moscato P (1989) On evolution, search, optimization, genetic algorithms and martial arts: towards memetic algorithms. Technical Report C3P Report 826, California Institute of Technology

  73. Nahvi B, Habibi J, Mohammadi K, Shamshirband S, Razgan OSA (2016) Using self-adaptive evolutionary algorithm to improve the performance of an extreme learning machine for estimating soil temperature. Comput Electron Agric 124:150–160

    Article  Google Scholar 

  74. Niu P, Ma Y, Li M, Yan S, Li G (2016) A kind of parameters self-adjusting extreme learning machine. Neural Process Lett 44(3):813–830

    Article  Google Scholar 

  75. Ojha VK, Abraham A, Snášel V (2017) Metaheuristic design of feedforward neural networks: a review of two decades of research. Eng Appl Artif Intell 60:97–116

    Article  Google Scholar 

  76. Salam MA, Hegazy O, Soliman OS (2015) Fpa-elm model for stock market prediction. Int J Adv Res Comput Sci Softw Eng 5(2):1050–1063

    Google Scholar 

  77. Pacifico LDS, Ludermir TB (2006) Evolutionary extreme learning machine based on particle swarm optimization and clustering strategies. In: International joint conference on neural networks (IJCNN). IEEE, pp 1–6

  78. Pacifico LDS, Ludermir TB (2013) Evolutionary extreme learning machine based on particle swarm optimization and clustering strategies. In: Neural networks (IJCNN), the 2013 international joint conference on. IEEE, pp 1–6

  79. Passino KM (2002) Biomimicry of bacterial foraging for distributed optimization and control. IEEE Control Syst 22(3):52–67

    Article  MathSciNet  Google Scholar 

  80. Qin-Yu Z, Qin AK, Suganthan PN, Huang G-B (2005) Evolutionary extreme learning machine. Pattern recognition 38(10):1759–1763

    Article  MATH  Google Scholar 

  81. Rosenblatt F (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev 65(6):386

    Article  Google Scholar 

  82. Rumelhart DE, Hinton GE, Williams RJ (1988) Neurocomputing: foundations of research chapter Learning Representations by back-propagating errors. MIT Press, Cambridge, pp 696–699

    Google Scholar 

  83. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning international representations by error propagation. In: Rumelhart DE, McCleland JL (eds) Parallel distributed processing: exploration in the microstructure of cognition, vol 1, Chap 8. MIT Press, Cambridge

    Google Scholar 

  84. Salam MA, Zawbaa HM, Emary E, Ghany KKA, Parv B (2016) A hybrid dragonfly algorithm with extreme learning machine for prediction. In: INnovations in Intelligent SysTems and Applications (INISTA), 2016 international symposium on. IEEE, pp 1–6

  85. Sánchez-Monedero J, Hervas-Martinez C, Gutiérrez PA, Ruz MC, Moreno MCR, Cruz-Ramirez M (2010) Evaluating the performance of evolutionary extreme learning machines by a combination of sensitivity and accuracy measures. Neural Netw World 20(7):899

    Google Scholar 

  86. Sattar AMA, Erturul ÖF, Gharabaghi B, McBean EA, Cao J (2017) Extreme learning machine model for water network management. J Neural Comput Appl 2017:1–13

    Google Scholar 

  87. Schuh MA, Angryk RA, Sheppard JW (2012) Evolving kernel functions with particle swarms and genetic programming. In: FLAIRS conference

  88. Schwefel H-P (1987) Collective phenomena in evolutionary systems

  89. Silva DNG, Pacifico LDS, Ludermir TB (2011) An evolutionary extreme learning machine based on group search optimization. In: Congress on evolutionary computation (CEC). IEEE, pp 574–580

  90. Storn R, Price K (1997) Differential evolutiona simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11(4):341–359

    Article  MATH  Google Scholar 

  91. Stützle TG (1999) Local search algorithms for combinatorial problems: analysis, improvements, and new applications, vol 220. Infix Sankt Augustin

  92. Sun X, Qin L (2014) An extreme learning machine based on quantum particle swarm optimization and its application in handwritten numeral recognition. In: Software engineering and service science (ICSESS), 2014 5th IEEE international conference on. IEEE, pp 323–326

  93. Sun Z-L, Choi T-M, Kin-Fan A, Yong Y (2008) Sales forecasting using extreme learning machine with applications in fashion retailing. Decis Support Syst 46(1):411–419

    Article  Google Scholar 

  94. Suresh S, Babu RV, Kim HJ (2009) No-reference image quality assessment using modified extreme learning machine classifier. Appl Soft Comput 9(2):541–552

    Article  Google Scholar 

  95. Tang J, Deng C, Huang G-B (2016) Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst 27(4):809–821

    Article  MathSciNet  Google Scholar 

  96. Voudouris C (1997) Guided local search for combinatorial optimisation problems. PhD Thesis, University of Essex

  97. Wang J, Ye K, Cao J, Wang T, Xue A, Cheng Y, Yin C (2017) Doa estimation of excavation devices with elm and music-based hybrid algorithm. Cogn Comput 9:1–17

    Article  Google Scholar 

  98. Wang Y, Cao F, Yuan Y (2011) A study on effectiveness of extreme learning machine. Neurocomputing 74(16):2483–2490

    Article  Google Scholar 

  99. Werbos PJ (1974) Beyond regression: new tools for prediction and analysis in the behavioral science. Ph. D. Thesis, Harvard University

  100. Wolpert DH, Macready WG (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1(1):67–82

    Article  Google Scholar 

  101. Xu X, Ji Z, Yuan F, Liu X (2014) A novel parallel approach of cuckoo search using mapreduce. In: 2014 international conference on computer, communications and information technology (CCIT 2014). Atlantis Press

  102. Xue B, Ma X, Gu J, Li Y (2013) An improved extreme learning machine based on variable-length particle swarm optimization. In: International conference on robotics and biomimetics (ROBIO). IEEE, pp 1030–1035

  103. Yang X-S (2010) Nature-inspired metaheuristic algorithms. Luniver Press, Bristol

    Google Scholar 

  104. You X, Shu Y (2006) Evolutionary extreme learning machine-based on particle swarm optimization. Adv Neural Netw ISNN 2006:644–652

    Google Scholar 

  105. Yang H, Yi J, Zhao J, Dong ZY (2013) Extreme learning machine based genetic algorithm and its application in power system economic dispatch. Neurocomputing 102:154–162

    Article  Google Scholar 

  106. Yang Z, Wen X, Wang Z (2015) Qpso-elm: an evolutionary extreme learning machine based on quantum-behaved particle swarm optimization. In: International conference on advanced computational intelligence (ICACI). IEEE, pp 69–72

  107. Yang Z, Zhang T, Zhang D (2016) A novel algorithm with differential evolution and coral reef optimization for extreme learning machine training. Cogn Neurodyn 10(1):73–83

    Article  MathSciNet  Google Scholar 

  108. Zhai J, Hong-yu X, Wang X (2012) Dynamic ensemble extreme learning machine based on sample entropy. Soft Comput 16(9):1493–1502

    Article  Google Scholar 

  109. Zhai J, Zang L, Zhou Z (2018) Ensemble dropout extreme learning machine via fuzzy integral for data classification. Neurocomputing 275:1043–1052

    Article  Google Scholar 

  110. Zhai J, Zhang S, Wang C (2017) The classification of imbalanced large data sets based on mapreduce and ensemble of elm classifiers. Int J Mach Learn Cybern 8(3):1009–1017

    Article  Google Scholar 

  111. Zhang Y, Cai Z, Wu J, Wang X, Liu X (2015) A memetic algorithm based extreme learning machine for classification. In: Neural networks (IJCNN), 2015 international joint conference on neural networks. IEEE, pp 1–8

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hossam Faris.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Eshtay, M., Faris, H. & Obeid, N. Metaheuristic-based extreme learning machines: a review of design formulations and applications. Int. J. Mach. Learn. & Cyber. 10, 1543–1561 (2019). https://doi.org/10.1007/s13042-018-0833-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-018-0833-6

Keywords

Navigation