Nonlinear Dynamics

, Volume 84, Issue 2, pp 895–914 | Cite as

Artificial neural network regression as a local search heuristic for ensemble strategies in differential evolution

  • Iztok Fister
  • Ponnuthurai Nagaratnam Suganthan
  • Iztok FisterJr.
  • Salahuddin M. Kamal
  • Fahad M. Al-Marzouki
  • Matjaž Perc
  • Damjan Strnad
Original Paper


Nature frequently serves as an inspiration for developing new algorithms to solve challenging real-world problems. Mathematical modeling has led to the development of artificial neural networks (ANNs), which have proven especially useful for solving problems such as classification and regression. Moreover, evolutionary algorithms (EAs), inspired by Darwin’s natural evolution, have been successfully applied to solve optimization, modeling, and simulation problems. Differential evolution (DE) is a particularly well-known EA that possesses a multitude of strategies for generating an offspring solution, where the best strategy is not known in advance. In this paper, the ANN regression is applied as a local search heuristic within the DE algorithm that tries predicting the best strategy or attempting to generate a better offspring from an ensemble of DE strategies. This local search heuristic is applied to the population of solutions according to a control parameter that regulates between the time complexity of calculation and the quality of the solution. The experiments on a CEC 2014 test suite consisting of 30 benchmark functions reveal the full potential in developing this idea.


Nonlinear dynamics Artificial neural network Differential evolution Regression Local search Ensemble strategies 



This research was supported by the Slovenian Research Agency (Grant P5-0027) and by the Deanship of Scientific Research, King Abdulaziz University (Grant 76-130-35-HiCi).


  1. 1.
    Adeyemo, J., Otieno, F.: Differential evolution algorithm for solving multi-objective crop planning model. Agric. Water Manag. 97(6), 848–856 (2010)CrossRefGoogle Scholar
  2. 2.
    Bhattacharya, A., Chattopadhyay, P.K.: Solving economic emission load dispatch problems using hybrid differential evolution. Appl. Soft Comput. 11(2), 2526–2537 (2011)CrossRefGoogle Scholar
  3. 3.
    Bigus, J.P.: Data Mining with Neural Networks: Solving Business Problems from Application Development to Decision Support. McGraw-Hill, Inc., New York (1996)Google Scholar
  4. 4.
    Brest, J., Greiner, S., Boskovic, B., Mernik, M., Zumer, V.: Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. IEEE Trans. Evolut. Comput. 10(6), 646–657 (2006)CrossRefGoogle Scholar
  5. 5.
    Cai, Y., Wang, J.: Differential evolution with neighborhood and direction information for numerical optimization. IEEE Trans. Cybern. 43(6), 2202–2215 (2013)CrossRefGoogle Scholar
  6. 6.
    Chakraborty, U.K., Das, S., Konar, A.: Differential evolution with local neighborhood. In: IEEE Congress on Evolutionary Computation, pp. 2042–2049. IEEE (2006)Google Scholar
  7. 7.
    Darwin, C.: On the Origin of Species. Harvard University Press, London (1859)Google Scholar
  8. 8.
    Das, S., Suganthan, P.: Differential evolution: a survey of the state-of-the-art. IEEE Trans. Evolut. Comput. 15(1), 4–31 (2011)CrossRefGoogle Scholar
  9. 9.
    Das, S., Abraham, A., Chakraborty, U.K., Konar, A.: Differential evolution using a neighborhood-based mutation operator. IEEE Trans. Evolut. Comput. 13(3), 526–553 (2009)CrossRefGoogle Scholar
  10. 10.
    Datta, D., Dutta, S.: A binary-real-coded differential evolution for unit commitment problem. Int. J. Electr. Power Energy Syst. 42(1), 517–524 (2012)CrossRefGoogle Scholar
  11. 11.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Elsayed, S.M., Sarker, R.A., Essam, D.L.: Multi-operator based evolutionary algorithms for solving constrained optimization problems. Comput. Oper. Res. 38(12), 1877–1896 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Fan, H.Y., Lampinen, J.: A trigonometric mutation operation to differential evolution. J. Global Optim. 27(1), 105–129 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Fister, I.J., Suganthan, P.N., Strnad, D., Brest, J., Fister, I.: Artificial neural network regression on ensemble strategies in differential evolution. In: MENDEL 2014, 20th International Conference on Soft Computing. University of Technology, Brno (2014)Google Scholar
  15. 15.
    Fister, I., Fister Jr, I., Yang, X.S., Brest, J.: A comprehensive review of firefly algorithms. Swarm Evolut. Comput. 13, 34–46 (2013)CrossRefGoogle Scholar
  16. 16.
    Fister, I., Rauter, S., Yang, X.S., Ljubič, K., Fister Jr, I.: Planning the sports training sessions with the bat algorithm. Neurocomputing 149(Part B), 993–1002 (2015)CrossRefGoogle Scholar
  17. 17.
    Fister Jr, I., Fister, D., Fister, I.: Differential evolution strategies with random forest regression in the bat algorithm. In: Proceeding of the Fifteenth Annual Conference Companion on Genetic and Evolutionary Computation Conference Companion 2013, pp. 1703–1706. ACM (2013)Google Scholar
  18. 18.
    Fister Jr, I., Fister, D., Yang, X.S.: A hybrid bat algorithm. Elektroteh. vestnik 80(1–2), 1–7 (2013)zbMATHGoogle Scholar
  19. 19.
    Friedman, M.: A comparison of alternative tests of significance for the problem of m rankings. Ann. Math. Stat. 11, 86–92 (1940)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Garro, B.A., Sossa, H., Vázquez, R.A.: Design of artificial neural networks using differential evolution algorithm. In: Wong, K.W., Mendis, B.S.U., Bouzerdoum, A. (eds.) Neural Information Processing. Models and Applications, pp. 201–208. Springer, Heidelberg (2010)Google Scholar
  21. 21.
    Gershenson, C.: Artificial Neural Networks for Beginners (2003). arXiv:cs/0308031
  22. 22.
    Han, M.F., Liao, S.H., Chang, J.Y., Lin, C.T.: Dynamic group-based differential evolution using a self-adaptive strategy for global optimization problems. Appl. Intell. 39(1), 41–56 (2013)CrossRefGoogle Scholar
  23. 23.
    Hecht-Nielsen, R.: Theory of the backpropagation neural network. In: International Joint Conference on Neural Networks, pp. 593–605. IEEE (1989)Google Scholar
  24. 24.
    Holger, H., Thomas, S.: Stochastic Local Search: Foundations & Applications. Morgan Kaufman Inc., Amsterdam (2004)zbMATHGoogle Scholar
  25. 25.
    Hozjan, T., Turk, G., Srpčič, S.: Fire analysis of steel frames with the use of artificial neural networks. J. Constr. Steel Res. 63(10), 1396–1403 (2007)CrossRefGoogle Scholar
  26. 26.
    J.J. Liang, B.Y.Q., Suganthan, P.N.: Problem definitions and evaluation criteria for the CEC 2014 special session and competition on single objective real-parameter numerical optimization. Technical Report, Zhengzhou University and Nanyang Technological University (2013)Google Scholar
  27. 27.
    Kartam, N., Flood, I., Garrett, J.H.: Artificial Neural Networks for Civil Engineers: Fundamentals and Applications. American Society of Civil Engineers, New York (1997)Google Scholar
  28. 28.
    Kourentzes, N., Barrow, D.K., Crone, S.F.: Neural network ensemble operators for time series forecasting. Expert Syst. Appl. 41(9), 4235–4244 (2014)CrossRefGoogle Scholar
  29. 29.
    LaTorre, A., Muelas, S., Peña, J.M.: A mos-based dynamic memetic differential evolution algorithm for continuous optimization: a scalability test. Soft Comput. 15(11), 2187–2199 (2011)CrossRefGoogle Scholar
  30. 30.
    Lee, S., Choeh, J.Y.: Predicting the helpfulness of online reviews using multilayer perceptron neural networks. Expert Syst. Appl. 41(6), 3041–3046 (2014)CrossRefGoogle Scholar
  31. 31.
    Lin, Y.C., Hwang, K.S., Wang, F.S.: Co-evolutionary hybrid differential evolution for mixed-integer optimization problems. Eng. Optim. 33(6), 663–682 (2001)CrossRefGoogle Scholar
  32. 32.
    Mallipeddi, R., Suganthan, P.: Differential evolution algorithm with ensemble of populations for global numerical optimization. Opsearch 46(2), 184–213 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Mallipeddi, R., Mallipeddi, S., Suganthan, P.N.: Ensemble strategies with adaptive evolutionary programming. Inf. Sci. 180(9), 1571–1581 (2010)CrossRefzbMATHGoogle Scholar
  34. 34.
    Mallipeddi, R., Suganthan, P.N., Pan, Q.K., Tasgetiren, M.F.: Differential evolution algorithm with ensemble of parameters and mutation strategies. Appl. Soft Comput. 11(2), 1679–1696 (2011)CrossRefGoogle Scholar
  35. 35.
    Mallipeddi, R., Suganthan, P.N.: Differential evolution algorithm with ensemble of parameters and mutation and crossover strategies. In: Panigrahi, B.K., Das, S., Suganthan, P.N., Dash, S.S. (eds.) Swarm, Evolutionary, and Memetic Computing, pp. 71–78. Springer, Heidelberg (2010)Google Scholar
  36. 36.
    Mallipeddi, R., Suganthan, P.N.: Ensemble differential evolution algorithm for CEC2011 problems. In: IEEE Congress on Evolutionary Computation, pp. 1557–1564. IEEE (2011)Google Scholar
  37. 37.
    McCulloch, W.S., Pitts, W.: A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5(4), 115–133 (1943)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Mininno, E., Neri, F., Cupertino, F., Naso, D.: Compact differential evolution. IEEE Trans. Evolut. Comput. 15(1), 32–54 (2011)CrossRefGoogle Scholar
  39. 39.
    Neri, F., Iacca, G., Mininno, E.: Disturbed exploitation compact differential evolution for limited memory optimization problems. Inf. Sci. 181(12), 2469–2487 (2011)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Neri, F., Mininno, E.: Memetic compact differential evolution for Cartesian robot control. IEEE Comput. Intell. Mag. 5(2), 54–65 (2010)CrossRefGoogle Scholar
  41. 41.
    Piotrowski, A.P.: Adaptive memetic differential evolution with global and local neighborhood-based mutation operators. Inf. Sci. 241, 164–194 (2013)CrossRefGoogle Scholar
  42. 42.
    Qin, A.K., Suganthan, P.N.: Self-adaptive differential evolution algorithm for numerical optimization. In: IEEE Congress on Evolutionary Computation, vol. 2, pp. 1785–1791. IEEE (2005)Google Scholar
  43. 43.
    Rahnamayan, S., Tizhoosh, H.R., Salama, M.M.: Opposition-based differential evolution. IEEE Trans. Evolut. Comput. 12(1), 64–79 (2008)CrossRefGoogle Scholar
  44. 44.
    Rocca, P., Oliveri, G., Massa, A.: Differential evolution as applied to electromagnetics. IEEE Antennas Propag. Mag. 53(1), 38–49 (2011)CrossRefGoogle Scholar
  45. 45.
    Rojas, R.: Neutral Networks: A Systematic Introduction. Springer, Berlin (1996)Google Scholar
  46. 46.
    Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach, 3rd edn. Prentice Hall, Englewood Cliffs (2010)zbMATHGoogle Scholar
  47. 47.
    Santos, J., Diéguez, M.: Differential evolution for protein structure prediction using the HP model. In: Ferrández, J.M., Álvarez, J.R., de la Paz, F., Toledo, F.J. (eds.) Foundations on Natural and Artificial Computation, pp. 323–333. Springer, Heidelberg (2011)Google Scholar
  48. 48.
    Storn, R., Price, K.: Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J. Global Optim. 11(4), 341–359 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Tvrdìk, J.: Competitive differential evolution. In: MENDEL 2006. 12th international conference on soft computing, pp. 7–12. University of Technology, Brno (2006)Google Scholar
  50. 50.
    Tvrdìk, J.: Differential evolution with competitive setting of its control parameters. TASK Q. 11, 169–179 (2007)Google Scholar
  51. 51.
    Tvrdìk, J.: Adaptation in differential evolution: a numerical comparison. Appl. Soft Comput. 9, 1149–1155 (2009)CrossRefGoogle Scholar
  52. 52.
    Uyar, A.Ş., Türkay, B., Keleş, A.: A novel differential evolution application to short-term electrical power generation scheduling. Int. J. Electr. Power Energy Syst. 33(6), 1236–1242 (2011)CrossRefGoogle Scholar
  53. 53.
    Vrugt, J.A., Robinson, B.A., Hyman, J.M.: Self-adaptive multimethod search for global optimization in real-parameter spaces. IEEE Trans. Evolut. Comput. 13(2), 243–259 (2009)CrossRefGoogle Scholar
  54. 54.
    Widrow, B., Rumelhart, D.E., Lehr, M.A.: Neural networks: applications in industry, business and science. Commun. ACM 37(3), 93–105 (1994)CrossRefGoogle Scholar
  55. 55.
    Zhang, W.J., Xie, X.F., et al.: DEPSO: hybrid particle swarm with differential evolution operator. IEEE Int. Conf. Syst. Man Cybern. 4, 3816–3821 (2003)Google Scholar
  56. 56.
    Zobaa, A., Reljin, B.: Neural network applications in electrical engineering. Neurocomputing 70(16–18), 2613–2614 (2007)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Iztok Fister
    • 1
  • Ponnuthurai Nagaratnam Suganthan
    • 2
  • Iztok FisterJr.
    • 1
  • Salahuddin M. Kamal
    • 3
  • Fahad M. Al-Marzouki
    • 3
  • Matjaž Perc
    • 3
    • 4
  • Damjan Strnad
    • 1
  1. 1.Faculty of Electrical Engineering and Computer ScienceUniversity of MariborMariborSlovenia
  2. 2.School of Electrical and Electronic EngineeringNanyang Technological UniversitySingaporeSingapore
  3. 3.Department of Physics, Faculty of SciencesKing Abdulaziz UniversityJeddahSaudi Arabia
  4. 4.Faculty of Natural Sciences and MathematicsUniversity of MariborMariborSlovenia

Personalised recommendations