Skip to main content
SpringerLink
Log in

Evolving neural networks using bird swarm algorithm for data classification and regression applications

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

This work proposes a new evolutionary multilayer perceptron neural networks using the recently proposed Bird Swarm Algorithm. The problem of finding the optimal connection weights and neuron biases is first formulated as a minimization problem with mean square error as the objective function. The BSA is then used to estimate the global optimum for this problem. A comprehensive comparative study is conducted using 13 classification datasets, three function approximation datasets, and one real-world case study (Tennessee Eastman chemical reactor problem) to benchmark the performance of the proposed evolutionary neural network. The results are compared with well-regarded conventional and evolutionary trainers and show that the proposed method provides very competitive results. The paper also considers a deep analysis of the results, revealing the flexibility, robustness, and reliability of the proposed trainer when applied to different datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://archive.ics.uci.edu/ml/

References

  1. Adwan, O., Faris, H., Jaradat, K., Harfoushi, O., Ghatasheh, N.: Predicting customer churn in telecom industry using multilayer preceptron neural networks: modeling and analysis. Life Sci. J. 11(3), 75–81 (2014)

    Google Scholar 

  2. Al-Hiary, H., Sheta, A., Ayesh, A.: Identification of a chemical process reactor using soft computing techniques. In: Proceedings of the 2008 International Conference on Fuzzy Systems (FUZZ2008) within the 2008 IEEE World Congress on Computational Intelligence (WCCI2008), Hong Kong, 1–6 June, pp. 845–653 (2008)

  3. Al-Shayea, Q.K.: Artificial neural networks in medical diagnosis. Int. J. Comput. Sci. Issues 8(2), 150–154 (2011)

    Google Scholar 

  4. Alboaneen, D.A., Tianfield, H., Zhang, Y.: Glowworm swarm optimisation for training multi-layer perceptrons. In: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, BDCAT ’17, pp. 131–138, New York, NY (2017). ACM

  5. Aljarah, I., Ludwig, S.A.: A mapreduce based glowworm swarm optimization approach for multimodal functions. In: 2013 IEEE Symposium on Swarm Intelligence (SIS), pp. 22–31. IEEE (2013)

  6. Aljarah, I., Ludwig, S.A.: Towards a scalable intrusion detection system based on parallel pso clustering using MapReduce. In: Proceedings of the 15th Annual Conference Companion on Genetic and Evolutionary Computation, pp. 169–170. ACM (2013)

  7. Aljarah, I., Ludwig, S.A.: A scalable mapreduce-enabled glowworm swarm optimization approach for high dimensional multimodal functions. Int. J. Swarm Intell. Res. (IJSIR) 7(1), 32–54 (2016)

    Google Scholar 

  8. Aljarah, I., Faris, H., Mirjalili, S.: Optimizing connection weights in neural networks using the whale optimization algorithm. Soft Comput. 22(1), 1–15 (2018)

    Google Scholar 

  9. Aljarah, I., Faris, H., Mirjalili, S., Al-Madi, N.: Training radial basis function networks using biogeography-based optimizer. Neural Comput. Appl. 29(7), 529–553 (2018)

    Google Scholar 

  10. Amaldi, E., Mayoraz, E., de Werra, D.: A review of combinatorial problems arising in feedforward neural network design. Discret. Appl. Math. 52(2), 111–138 (1994)

    MathSciNet  MATH  Google Scholar 

  11. Arifovic, J., Gencay, R.: Using genetic algorithms to select architecture of a feedforward artificial neural network. Physica A 289(3), 574–594 (2001)

    MATH  Google Scholar 

  12. Barton, I.P., Martinsen, S.W.: Equation-oriented simulator training. In Proceedings of the American Control Conference, Albuquerque, New Mexico, pp. 2960–2965 (1997)

  13. Basheer, I.A., Hajmeer, M.: Artificial neural networks: fundamentals, computing, design, and application. J. Microbiol. Methods 43(1), 3–31 (2000)

    Google Scholar 

  14. Bathelt, A., Ricker, N.L., Jelali, M.: Revision of the Tennessee Eastman process model. IFAC-PapersOnLine 48(8), 309–314 (2015)

    Google Scholar 

  15. Bebis, G., Georgiopoulos, M.: Feed-forward neural networks. IEEE Potentials 13(4), 27–31 (1994)

    Google Scholar 

  16. Bhat, N., McAvoy, T.J.: Use of neural nets for dynamic modeling and control of chemical process systems. Comput. Chem. Eng. 14, 573–582 (1990)

    Google Scholar 

  17. Bornholdt, S., Graudenz, D.: General asymmetric neural networks and structure design by genetic algorithms. Neural Netw. 5(2), 327–334 (1992)

    Google Scholar 

  18. BoussaïD, I., Lepagnot, J., Siarry, P.: A survey on optimization metaheuristics. Inf. Sci. 237, 82–117 (2013)

    MathSciNet  MATH  Google Scholar 

  19. Brajevic, I., Tuba, M.: Training feed-forward neural networks using firefly algorithm. In: Proceedings of the 12th International Conference on Artificial Intelligence, Knowledge Engineering and Data Bases (AIKED’13), pp. 156–161 (2013)

  20. Buscema, M.: Back propagation neural networks. Subst. Use Misuse 33(2), 233–270 (1998)

    Google Scholar 

  21. Chen, C.L.P.: A rapid supervised learning neural network for function interpolation and approximation. IEEE Trans. Neural Netw. 7(5), 1220–1230 (1996)

    MathSciNet  Google Scholar 

  22. Downs, J.J., Vogel, E.F.: A plant-wide industrial process control problem. Comput. Chem. Eng. 17(3), 245–255 (1993)

    Google Scholar 

  23. Engelbrecht, A.P.: Supervised learning neural networks. Computational Intelligence: An Introduction, 2nd edn., pp. 27-54. Wiley, Singapore (2007)

  24. Faris, H., Alkasassbeh, M., Rodan, A.: Artificial neural networks for surface ozone prediction: models and analysis. Pol. J. Environ. Stud. 23(2), 341–348 (2014)

    Google Scholar 

  25. Faris, H., Aljarah, I., et al.: Optimizing feedforward neural networks using krill herd algorithm for e-mail spam detection. In: 2015 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT), pp. 1–5. IEEE (2015)

  26. Faris, H., Aljarah, I., Al-Madi, N., Mirjalili, S.: Optimizing the learning process of feedforward neural networks using lightning search algorithm. Int. J. Artif. Intell. Tools 25(06), 1650033 (2016)

    Google Scholar 

  27. Faris, H., Aljarah, I., Mirjalili, S.: Training feedforward neural networks using multi-verse optimizer for binary classification problems. Applied Intelligence, pp. 1–11 (2016)

  28. Faris, H., Aljarah, I., Mirjalili, S.: Evolving radial basis function networks using moth–flame optimizer. In: Handbook of Neural Computation, pp. 537–550. Elsevier (2017)

  29. Faris, H., Aljarah, I., Mirjalili, S.: Improved monarch butterfly optimization for unconstrained global search and neural network training. Appl. Intell. 48(2), 445–464 (2018)

    Google Scholar 

  30. Galić, E., Höhfeld, M.: Improving the generalization performance of multi-layer-perceptrons with population-based incremental learning. In: International Conference on Parallel Problem Solving from Nature, pp. 740–750. Springer (1996)

  31. Garro, B.A., Vázquez, R.A.: Designing artificial neural networks using particle swarm optimization algorithms. Comput. Intell. Neurosci. https://doi.org/10.1155/2015/369298 (2015)

    Article  Google Scholar 

  32. Goerick, C., Rodemann, T.: Evolution strategies: an alternative to gradient-based learning. In: Proceedings of the International Conference on Engineering Applications of Neural Networks, vol. 1, pp. 479–482 (1996)

  33. Goldberg, D.E. et al.: Genetic Algorithms in Search Optimization and Machine Learning, vol. 412. Addison-Wesley, Reading (1989)

  34. Golfinopoulos, E., Tourville, J.A., Guenther, F.H.: The integration of large-scale neural network modeling and functional brain imaging in speech motor control. Neuroimage 52(3), 862–874 (2010)

    Google Scholar 

  35. Gupta, J.N.D., Sexton, R.S.: Comparing backpropagation with a genetic algorithm for neural network training. Omega 27(6), 679–684 (1999)

    Google Scholar 

  36. Gupta, M.M., Jin, L., Homma, N.: Radial basis function neural networks. In: Static and Dynamic Neural Networks: From Fundamentals to Advanced Theory, pp. 223–252 (2003)

  37. Hansel, D., Sompolinsky, H.: Learning from examples in a single-layer neural network. EPL Europhys. Lett. 11(7), 687 (1990)

    Google Scholar 

  38. Heidari, A.A., Faris, H., Aljarah, I., Mirjalili, S.: An efficient hybrid multilayer perceptron neural network with grasshopper optimization. Soft Comput. https://doi.org/10.1007/s00500-018-3424-2(2018)

    Article  Google Scholar 

  39. Ho, Y.-C., Pepyne, D.L.: Simple explanation of the no-free-lunch theorem and its implications. J. Optim. Theory Appl. 115(3), 549–570 (2002)

    MathSciNet  MATH  Google Scholar 

  40. Hush, D.R., Horne, B.G.: Progress in supervised neural networks. IEEE Signal Process. Mag. 10(1), 8–39 (1993)

    Google Scholar 

  41. Hwang, Y.-S., Bang, S.-Y.: An efficient method to construct a radial basis function neural network classifier. Neural Netw. 10(8), 1495–1503 (1997)

    Google Scholar 

  42. Ilonen, J., Kamarainen, J.-K., Lampinen, J.: Differential evolution training algorithm for feed-forward neural networks. Neural Process. Lett. 17(1), 93–105 (2003)

    Google Scholar 

  43. Juricek, B.C., Seborg, D.E., Larimore, W.E.: Identification of the tennessee eastman challenge process with subspace methods. Control Eng. Pract. 9(12), 1337–1351 (2001)

    Google Scholar 

  44. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)

    Google Scholar 

  45. Karaboga, D., Akay, B., Ozturk, C.: Artificial bee colony (abc) optimization algorithm for training feed-forward neural networks. In: International Conference on Modeling Decisions for Artificial Intelligence, pp. 318–329. Springer (2007)

  46. Karim, M.N., Rivera, S.L.: Artificial neural networks in bioprocess state estimation. Adv. Biochem. Eng. Biotechnol. 46, 1–31 (1992)

    Google Scholar 

  47. Khan, K., Sahai, A.: A comparison of ba, ga, pso, bp and lm for training feed forward neural networks in e-learning context. Int. J. Intell. Syst. Appl. 4(7), 23 (2012)

    Google Scholar 

  48. Kowalski, P.A., Łukasik, S.: Training neural networks with krill herd algorithm. Neural Process. Lett. 44, 5–17 (2015)

    Google Scholar 

  49. Kruse, R., Borgelt, C., Klawonn, F., Moewes, C., Steinbrecher, M., Held, P.: Multi-layer perceptrons. In: Computational Intelligence, pp. 47–81. Springer (2013)

  50. Larochelle, H., Bengio, Y., Louradour, J., Lamblin, P.: Exploring strategies for training deep neural networks. J. Mach. Learn. Res. 10, 1–40 (2009)

    MATH  Google Scholar 

  51. Leonard, J., Kramer, M.A.: Improvement of the Backpropagation algorithm for training neural networks. Comput. Chem. Eng. 14, 337–343 (1990)

    Google Scholar 

  52. Leshno, M., Lin, V.Y., Pinkus, A., Schocken, S.: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw. 6(6), 861–867 (1993)

    Google Scholar 

  53. Leung, F.H.-F., Lam, H.-K., Ling, S.-H., Tam, P.K.-S.: Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Trans. Neural Netw. 14(1), 79–88 (2003)

    Google Scholar 

  54. Lichman, M.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine (2013)

    Google Scholar 

  55. Lippmann, R.P.: Pattern classification using neural networks. IEEE Commun. Mag. 27(11), 47–50 (1989)

    Google Scholar 

  56. Lo, S.-C.B., Chan, H.-P., Lin, J.-S., Li, H., Freedman, M.T., Mun, S.K.: Artificial convolution neural network for medical image pattern recognition. Neural Netw. 8(7), 1201–1214 (1995)

    Google Scholar 

  57. Mavrovouniotis, M., Yang, S.: Training neural networks with ant colony optimization algorithms for pattern classification. Soft Comput. 19(6), 1511–1522 (2015)

    Google Scholar 

  58. Meissner, M., Schmuker, M., Schneider, G.: Optimized particle swarm optimization (OPSO) and its application to artificial neural network training. BMC Bioinform. 7(1), 125 (2006)

    Google Scholar 

  59. Melin, P., Castillo, O.: Unsupervised learning neural networks. In: Hybrid Intelligent Systems for Pattern Recognition Using Soft Computing, pp. 85–107. Springer (2005)

  60. Meng, X.-B., Gao, X.Z., Lu, L., Liu, Y., Zhang, H.: A new bio-inspired optimisation algorithm: bird swarm algorithm. J. Exp. Theor. Artif. Intell. https://doi.org/10.1080/0952813X.2015.1042530(2015)

    Article  Google Scholar 

  61. Merkl, D., Rauber, A.: Document classification with unsupervised artificial neural networks. In: Soft Computing in Information Retrieval, pp. 102–121. Springer (2000)

  62. Mezura-Montes, E., Velázquez-Reyes, J., Coello Coello, C.A.: A comparative study of differential evolution variants for global optimization. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 485–492. ACM (2006)

  63. Mirjalili, S.: How effective is the grey wolf optimizer in training multi-layer perceptrons. Appl. Intell. 43(1), 150–161 (2015)

    Google Scholar 

  64. Mirjalili, S., Mirjalili, S.M., Lewis, A.: Let a biogeography-based optimizer train your multi-layer perceptron. Inf. Sci. 269, 188–209 (2014)

    MathSciNet  Google Scholar 

  65. Mitchell, T.M: Artificial neural networks. Machine Learning, pp. 81–127 (1997)

  66. Montana, D.J., Davis, L.: Training feedforward neural networks using genetic algorithms. IJCAI 89, 762–767 (1989)

    MATH  Google Scholar 

  67. Nahas, E.P., Henson, M.A., Seborg, D.E.: Nonlinear internal model control strategy for neural network models. Comput. Chem. Eng. 16, 1039–1057 (1992)

    Google Scholar 

  68. Nawi, N.M., Khan, A., Rehman, M.Z., Tutut H., Mustafa, M.D.: Comparing performances of cuckoo search based neural networks. In: Recent Advances on Soft Computing and Data Mining, pp. 163–172. Springer (2014)

  69. Parisi, R., Di Claudio, E.D., Lucarelli, G., Orlandi, G.: Car plate recognition by neural networks and image processing. In: Proceedings of the 1998 IEEE International Symposium on Circuits and Systems, 1998. ISCAS’98, vol. 3, pp. 195–198. IEEE (1998)

  70. Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. ICML 3(28), 1310–1318 (2013)

    Google Scholar 

  71. Principe, J.C., Fancourt, C.L.: Artificial neural networks. In: Pardalos, P.M., Romejin, H.E. (eds.) Handbook of Global Optimization, vol. 2, pp. 363–386. Kluwer, Dordrecht (2013)

    Google Scholar 

  72. Ricker, N.L.: Nonlinear model predictive control of the tennessee eastman challenge process. Comput. Chem. Eng. 19(9), 961–981 (1995)

    Google Scholar 

  73. Ricker, N.L.: Nonlinear modeling and state estimation of the tennessee eastman challenge process. Comput. Chem. Eng. 19(9), 983–1005 (1995)

    Google Scholar 

  74. Ricker, N.L.: Tennessee Eastman challenge archive (2005)

  75. Sanger, T.D.: Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural Netw. 2(6), 459–473 (1989)

    Google Scholar 

  76. Seiffert, U.: Multiple layer perceptron training using genetic algorithms. In: ESANN, pp. 159–164. Citeseer (2001)

  77. Sheta, A., Al-Hiary, Heba, Braik, Malik: Identification and model predictive controller design of the Tennessee Eastman chemical process using ann. In: Proceedings of the 2009 International Conference on Artificial Intelligence (ICAI’09), July 13–16, USA, vol. 1, pp. 25–31 (2009)

  78. Sibi, P., Allwyn Jones, S., Siddarth, P.: Analysis of different activation functions using back propagation neural networks. J. Theor. Appl. Inf. Technol. 47(3), 1264–1268 (2013)

    Google Scholar 

  79. Simon, D.: Biogeography-based optimization. IEEE Trans. Evol. Comput. 12(6), 702–713 (2008)

    Google Scholar 

  80. Sivagaminathan, R.K., Ramakrishnan, S.: A hybrid approach for feature subset selection using neural networks and ant colony optimization. Expert Syst. Appl. 33(1), 49–60 (2007)

    Google Scholar 

  81. Slowik, A., Bialko, M.: Training of artificial neural networks using differential evolution algorithm. In: 2008 Conference on Human System Interactions, pp. 60–65. IEEE (2008)

  82. Socha, K., Blum, C.: An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training. Neural Comput. Appl. 16(3), 235–247 (2007)

    Google Scholar 

  83. Stanley, K.O.: Efficient reinforcement learning through evolving neural network topologies. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2002). Citeseer (2002)

  84. Subudhi, B., Jena, D.: Differential evolution and Levenberg Marquardt trained neural network scheme for nonlinear system identification. Neural Process. Lett. 27(3), 285–296 (2008)

    Google Scholar 

  85. Suykens, J.A.K., Vandewalle, J.P.L., de Moor, B.L.: Artificial Neural Networks for Modelling and Control of Non-linear Systems. Springer, Berlin (2012)

    Google Scholar 

  86. Valian, E., Mohanna, S., Tavakoli, S.: Improved cuckoo search algorithm for feedforward neural network training. Int. J. Artif. Intell. Appl. 2(3), 36–43 (2011)

    Google Scholar 

  87. van den Bergh, F., Engelbrecht, A.P., Engelbrecht, A.P.: Cooperative learning in neural networks using particle swarm optimizers. In: South African Computer Journal. Citeseer (2000)

  88. Wdaa, A.S.I.: Differential evolution for neural networks learning enhancement. PhD thesis, Universiti Teknologi Malaysia (2008)

  89. Whitley, D., Starkweather, T., Bogart, C.: Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput. 14(3), 347–361 (1990)

    Google Scholar 

  90. Wienholt, W.: Minimizing the system error in feedforward neural networks with evolution strategy. In: ICANN’93, pp. 490–493. Springer (1993)

  91. Yamany, W., Fawzy, M., Tharwat, A., Hassanien, A.E.: Moth-flame optimization for training multi-layer perceptrons. In: 2015 11th International Computer Engineering Conference (ICENCO), pp. 267–272. IEEE (2015)

  92. Yang, C.C., Prasher, S.O., Landry, J.A., DiTommaso, A.: Application of artificial neural networks in image recognition and classification of crop and weeds. Can. Agric. Eng. 42(3), 147–152 (2000)

    Google Scholar 

  93. Yang, Z., Hoseinzadeh, M., Andrews, A., Mayers, C., Evans, D.T., Bolt, R.T., Bhimani, J., Mi, N., Swanson, S.: Autotiering: automatic data placement manager in multi-tier all-flash datacenter. In: 2017 IEEE 36th International on Performance Computing and Communications Conference (IPCCC), pp. 1–8. IEEE (2017)

  94. Yang, Z., Jia, D., Ioannidis, S., Mi, N., Sheng, B.: Intermediate data caching optimization for multi-stage and parallel big data frameworks. arXiv:1804.10563 (2018)

  95. Yao, X.: A review of evolutionary artificial neural networks. Int. J. Intell. Syst. 8(4), 539–567 (1993)

    Google Scholar 

  96. Yegnanarayana, B.: Artificial neural networks. PHI Learning Pvt. Ltd., New Delhi (2009)

    Google Scholar 

  97. Zhang, G.P.: Neural networks for classification: a survey. IEEE Trans. Syst. Man Cybern. C 30(4), 451–462 (2000)

    MathSciNet  Google Scholar 

  98. Zhang, N.: An online gradient method with momentum for two-layer feedforward neural networks. Appl. Math. Comput. 212(2), 488–498 (2009)

    MathSciNet  MATH  Google Scholar 

  99. Zhang, C., Shao, H., Li, Y.: Particle swarm optimisation for evolving artificial neural network. In: 2000 IEEE International Conference on Systems, Man, and Cybernetics, vol. 4, pp. 2487–2490. IEEE (2000)

  100. Zhang, J., Sanderson, A.C.: Jade: adaptive differential evolution with optional external archive. IEEE Trans. Evol. Comput. 13(5), 945–958 (2009)

    Google Scholar 

  101. Zhang, J.-R., Zhang, J., Lok, T.-M., Lyu, M.R.: A hybrid particle swarm optimization–back-propagation algorithm for feedforward neural network training. Appl. Math. Comput. 185(2), 1026–1037 (2007)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, King Abdullah II School for Information Technology, The University of Jordan, Amman, Jordan

    Ibrahim Aljarah & Hossam Faris

  2. School of Information and Communication Technology, Griffith University, Nathan, Brisbane, QLD, 4111, Australia

    Seyedali Mirjalili

  3. King Hussein Faculty of Computing Sciences, Princess Sumaya University for Technology, Amman, Jordan

    Nailah Al-Madi

  4. Department of Computing Sciences, Texas A&M University, Corpus Christi, TX, 78412, USA

    Alaa Sheta

  5. Department of Computer Science, Birzeit University, Birzeit, Palestine

    Majdi Mafarja

Authors
  1. Ibrahim Aljarah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hossam Faris
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seyedali Mirjalili
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nailah Al-Madi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alaa Sheta
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Majdi Mafarja
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ibrahim Aljarah.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aljarah, I., Faris, H., Mirjalili, S. et al. Evolving neural networks using bird swarm algorithm for data classification and regression applications. Cluster Comput 22, 1317–1345 (2019). https://doi.org/10.1007/s10586-019-02913-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-019-02913-5

Keywords

Search

Navigation