Advertisement

Nature Inspired Meta-heuristic Algorithms for Deep Learning: Recent Progress and Novel Perspective

  • Haruna ChiromaEmail author
  • Abdulsalam Ya’u Gital
  • Nadim Rana
  • Shafi’i M. Abdulhamid
  • Amina N. Muhammad
  • Aishatu Yahaya Umar
  • Adamu I. AbubakarEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 943)

Abstract

Deep learning is presently attracting extra ordinary attention from both the industry and the academia. The application of deep learning in computer vision has recently gain popularity. The optimization of deep learning models through nature inspired algorithms is a subject of debate in computer science. The application areas of the hybrid of natured inspired algorithms and deep learning architecture includes: machine vision and learning, image processing, data science, autonomous vehicles, medical image analysis, biometrics, etc. In this paper, we present recent progress on the application of nature inspired algorithms in deep learning. The survey pointed out recent development issues, strengths, weaknesses and prospects for future research. A new taxonomy is created based on natured inspired algorithms for deep learning. The trend of the publications in this domain is depicted; it shows the research area is growing but slowly. The deep learning architectures not exploit by the nature inspired algorithms for optimization are unveiled. We believed that the survey can facilitate synergy between the nature inspired algorithms and deep learning research communities. As such, massive attention can be expected in a near future.

Keywords

Deep learning Deep belief network Cuckoo search algorithm Convolutional neural network Firefly algorithm Nature inspired algorithms 

References

  1. 1.
    McCulloch, W.S., Pitts, W.: A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5, 115–133 (1943)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Holland, J.: Adaptation in natural and artificial systems: an introductory analysis with application to biology, Control and artificial intelligence (1975)Google Scholar
  3. 3.
    Yang, X.-S., Deb, S.: Cuckoo search via lévy flights. In: 2009 World Congress on Nature and Biologically Inspired Computing, NaBIC 2009, pp. 210–214 (2009)Google Scholar
  4. 4.
    Karaboga, D.: An idea based on honey bee swarm for numerical optimization, Technical report-tr06, Erciyes university, engineering faculty, computer engineering department (2005)Google Scholar
  5. 5.
    Yang, X.-S., Deb, S., Fong, S., He, X., Zhao, Y.: Swarm intelligence: today and tomorrow. In: 2016 3rd International Conference on Soft Computing and Machine Intelligence (ISCMI), pp. 219–223 (2016)Google Scholar
  6. 6.
    Chiroma, H., Abdul-kareem, S., Ibrahim, U., Ahmad, I.G., Garba, A., Abubakar, A., et al.: Malaria severity classification through Jordan-Elman neural network based on features extracted from thick blood smear. Neural Netw. World 25, 565 (2015)CrossRefGoogle Scholar
  7. 7.
    Chaoui, H., Ibe-Ekeocha, C.C.: State of charge and state of health estimation for lithium batteries using recurrent neural networks. IEEE Trans. Veh. Technol. 66, 8773–8783 (2017)CrossRefGoogle Scholar
  8. 8.
    Dolezel, P., Skrabanek, P., Gago, L.: Pattern recognition neural network as a tool for pest birds detection. In: 2016 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–6 (2016)Google Scholar
  9. 9.
    Nie, L., Guan, J., Lu, C., Zheng, H., Yin, Z.: Longitudinal speed control of autonomous vehicle based on a self-adaptive PID of radial basis function neural network. IET Intel. Transp. Syst. 12(6), 485–494 (2018)CrossRefGoogle Scholar
  10. 10.
    Bahrammirzaee, A.: A comparative survey of artificial intelligence applications in finance: artificial neural networks, expert system and hybrid intelligent systems. Neural Comput. Appl. 19, 1165–1195 (2010)CrossRefGoogle Scholar
  11. 11.
    Xu, Y., Cheng, J., Wang, L., Xia, H., Liu, F., Tao, D.: Ensemble one-dimensional convolution neural networks for skeleton-based action recognition. IEEE Sig. Process. Lett. 25(7), 1044–1048 (2018)CrossRefGoogle Scholar
  12. 12.
    Lam, H., Ling, S., Leung, F.H., Tam, P.K.-S.: Tuning of the structure and parameters of neural network using an improved genetic algorithm. In: 2001 The 27th Annual Conference of the IEEE Industrial Electronics Society, IECON 2001, pp. 25–30 (2001)Google Scholar
  13. 13.
    Chiroma, H., Abdulkareem, S., Abubakar, A., Herawan, T.: Neural networks optimization through genetic algorithm searches: a review. Appl. Math. 11, 1543–1564 (2017)Google Scholar
  14. 14.
    Karaboga, D., Akay, B., Ozturk, C.: Artificial bee colony (ABC) optimization algorithm for training feed-forward neural networks. In: International Conference on Modeling Decisions for Artificial Intelligence 2007, pp. 318–329 (2007)Google Scholar
  15. 15.
    Nawi, N.M., Khan, A., Rehman, M.Z.: A new back-propagation neural network optimized with cuckoo search algorithm. In: International Conference on Computational Science and Its Applications 2013, pp. 413–426 (2013)Google Scholar
  16. 16.
    Juang, C.-F.: A hybrid of genetic algorithm and particle swarm optimization for recurrent network design. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 34, 997–1006 (2004)CrossRefGoogle Scholar
  17. 17.
    Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Fong, S., Deb, S., Yang, X.-s.: How meta-heuristic algorithms contribute to deep learning in the hype of big data analytics. In: Progress in Intelligent Computing Techniques: Theory, Practice, and Applications, pp. 3–25. Springer (2018)Google Scholar
  19. 19.
    Papa, J.P., Rosa, G.H., Pereira, D.R., Yang, X.-S.: Quaternion-based deep belief networks fine-tuning. Appl. Soft Comput. 60, 328–335 (2017)CrossRefGoogle Scholar
  20. 20.
    Fister Jr., I., Yang, X.-S., Fister, I., Brest, J., Fister, D.: A brief review of nature-inspired algorithms for optimization, arXiv preprint arXiv:1307.4186 (2013)
  21. 21.
    Xing, B., Gao, W.-J.: Innovative computational intelligence: a rough guide to 134 clever algorithms. Springer (2014)Google Scholar
  22. 22.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436 (2015)CrossRefGoogle Scholar
  23. 23.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intel. 35, 1798–1828 (2013)CrossRefGoogle Scholar
  24. 24.
    Zhang, C., Tan, K.C., Li, H., Hong, G.S.: A cost-sensitive deep belief network for imbalanced classification. IEEE Trans. Neural Netw. Learn. Syst. 28(99), 1–4 (2018)Google Scholar
  25. 25.
    Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., Martinez-Gonzalez, P., Garcia-Rodriguez, J.: A survey on deep learning techniques for image and video semantic segmentation. Appl. Soft. Comput. 70, 41–65 (2018)CrossRefGoogle Scholar
  26. 26.
    Liu, Y., Chen, X., Wang, Z., Wang, Z.J., Ward, R.K., Wang, X.: Deep learning for pixel-level image fusion: recent advances and future prospects. Inf. Fusion 42, 158–173 (2018)CrossRefGoogle Scholar
  27. 27.
    Yaseen, M.U., Anjum, A., Rana, O., Antonopoulos, N.: Deep learning hyper-parameter optimization for video analytics in clouds. IEEE Trans. Syst. Man Cybern: Syst 15(99), 1–12 (2018)Google Scholar
  28. 28.
    Neterer, J.R., Guzide, O.: Deep learning in natural language processing. Proc. West Va. Acad. Sci. 90(1) (2018)Google Scholar
  29. 29.
    Tang, M., Gao, H., Zhang, Y., Liu, Y., Zhang, P., Wang, P.: Research on deep learning techniques in breaking text-based captchas and designing image-based captcha. IEEE Trans. Inf. Forensics Secur. 13, 2522–2537 (2018)CrossRefGoogle Scholar
  30. 30.
    Pathak, A.R., Pandey, M., Rautaray, S.: Application of deep learning for object detection. Proc. Comput. Sci. 132, 1706–1717 (2018)CrossRefGoogle Scholar
  31. 31.
    Ji, Y., Liu, L., Wang, H., Liu, Z., Niu, Z., Denby, B.: Updating the Silent Speech Challenge benchmark with deep learning. Speech Commun. 98, 42–50 (2018)CrossRefGoogle Scholar
  32. 32.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Papa, J.P., Rosa, G.H., Marana, A.N., Scheirer, W., Cox, D.D.: Model selection for discriminative restricted boltzmann machines through meta-heuristic techniques. J. Comput. Sci. 9, 14–18 (2015)CrossRefGoogle Scholar
  34. 34.
    Papa, J.P., Rosa, G.H., Costa, K.A., Marana, N.A., Scheirer, W., Cox, D.D.: On the model selection of bernoulli restricted Boltzmann machines through harmony search. In: Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 1449–1450 (2015)Google Scholar
  35. 35.
    Rosa, G., Papa, J., Costa, K., Passos, L., Pereira, C., Yang, X.-S.: Learning parameters in deep belief networks through firefly algorithm. In: IAPR Workshop on Artificial Neural Networks in Pattern Recognition, pp. 138–149 (2016)CrossRefGoogle Scholar
  36. 36.
    Rodrigues, D., Yang, X.-S., Papa, J.: Fine-tuning deep belief networks using cuckoo search. In: Bio-Inspired Computation and Applications in Image Processing, pp. 47–59 (2017)CrossRefGoogle Scholar
  37. 37.
    Ma, M., Sun, C., Chen, X.: Discriminative deep belief networks with ant colony optimization for health status assessment of machine. IEEE Trans. Instrum. Measur. 66, 3115–3125 (2017)CrossRefGoogle Scholar
  38. 38.
    Kuremoto, T., Kimura, S., Kobayashi, K., Obayashi, M.: Time series forecasting using restricted Boltzmann machine. In: International Conference on Intelligent Computing, pp. 17–22 (2012)Google Scholar
  39. 39.
    Baldominos, A., Saez, Y., Isasi, P.: Evolutionary convolutional neural networks: An application to handwriting recognition. Neurocomputing 283, 38–52 (2018)CrossRefGoogle Scholar
  40. 40.
    Liu, K., Zhang, L.M., Sun, Y.W.: Deep Boltzmann machines aided design based on genetic algorithms. In: Applied Mechanics and Materials, pp. 848–851 (2014)Google Scholar
  41. 41.
    Levy, E., David, O.E., Netanyahu, N.S.: Genetic algorithms and deep learning for automatic painter classification. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 1143–1150 (2014)Google Scholar
  42. 42.
    Rere, L.R., Fanany, M.I., Arymurthy, A.M.: Simulated annealing algorithm for deep learning. Proc. Comput. Sci. 72, 137–144 (2015)CrossRefGoogle Scholar
  43. 43.
    Fedorovici, L.-O., Precup, R.-E., Dragan, F., David, R.-C., Purcaru, C.: Embedding gravitational search algorithms in convolutional neural networks for OCR applications. In: 7th IEEE International Symposium on Applied Computational Intelligence and Informatics (SACI) 2012, pp. 125–130 (2012)Google Scholar
  44. 44.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)CrossRefGoogle Scholar
  45. 45.
    Chaturvedi, I., Ong, Y.-S., Tsang, I.W., Welsch, R.E., Cambria, E.: Learning word dependencies in text by means of a deep recurrent belief network. Knowl.-Based Syst. 108, 144–154 (2016)CrossRefGoogle Scholar
  46. 46.
    Mannepalli, K., Sastry, P.N., Suman, M.: A novel adaptive fractional deep belief networks for speaker emotion recognition. Alexandria Eng. J. 56(4), 485–497 (2016)CrossRefGoogle Scholar
  47. 47.
    Qiao, J., Wang, G., Li, X., Li, W.: A self-organizing deep belief network for nonlinear system modeling. Appl. Soft Comput. 65, 170–183 (2018)CrossRefGoogle Scholar
  48. 48.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Haruna Chiroma
    • 1
    Email author
  • Abdulsalam Ya’u Gital
    • 2
  • Nadim Rana
    • 3
  • Shafi’i M. Abdulhamid
    • 4
  • Amina N. Muhammad
    • 5
  • Aishatu Yahaya Umar
    • 5
  • Adamu I. Abubakar
    • 6
    Email author
  1. 1.Department of Computer ScienceFederal College of Education (Technical)GombeNigeria
  2. 2.Department of Mathematical SciencesAbubakar Tafawa Balewa UniversityBauchiNigeria
  3. 3.College of Computer Science and Information SystemsJazan UniversityJazanKingdom of Saudi Arabia
  4. 4.Department of Cyber Security ScienceFederal University of TechnologyMinnaNigeria
  5. 5.Department of MathematicsGombe State UniversityGombeNigeria
  6. 6.Department of Computer ScienceInternational Islamic University MalaysiaGombakMalaysia

Personalised recommendations