Skip to main content
SpringerLink
Log in

A survey: evolutionary deep learning

  • Data analytics and machine learning
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Deep learning (DL) has made remarkable progress on various real-world tasks, but its construction pipeline strongly relies on human scientists. Furthermore, evolutionary computing (EC), as an optimization tool based on the biological evolution mechanism, has good performance on complex optimization problems. It provides a new way to construct DL models and has generated many sparks in the DL field, especially in automatic machine learning (AutoML). Although many reviews have been conducted on AutoML, in recent years, few comprehensive works have studied on the application of EC in DL, which is called evolutionary deep learning (EDL). After a thorough investigation, we think that EDL can be divided into four parts: (1) learning rule optimization, (2) hyperparameter optimization, (3) neural architecture search, and (4) other EDL-related works. In this work, we introduce the classic optimization methods and the challenges of EDL with respect to these four parts, review the related work, and then present the future research prospects. This work clearly and comprehensively reviews the concept and research content of EDL, which can help readers quickly find the intersection between EC and DL and seek their inspiration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

References

  • Abraham A (2004) Meta learning evolutionary artificial neural networks. Neurocomputing 56:1–38. https://doi.org/10.1016/S0925-2312(03)00369-2

    Article  Google Scholar 

  • Adya M, Collopy F (1998) How effective are neural networks at forecasting and prediction? A review and evaluation. J Forecast 17(5–6):481–495. https://doi.org/10.1002/ (SICI)1099-131X(1998090)17:5/6<481::AID-FOR709>3.0.CO;2-Q

  • Al-Sahaf H, Bi Y, Chen Q et al (2019) A survey on evolutionary machine learning. J R Soc N Z 49(2):205–228. https://doi.org/10.1080/03036758.2019.1609052

    Article  Google Scholar 

  • Alba E, Chicano JF (2004) Training neural networks with GA hybrid algorithms. In: Proceedings of the 2004 genetic and evolutionary computation conference, pp 852–863. https://doi.org/10.1007/978-3-540-24854-5_87

  • Alexandropoulos SAN, Aridas CK, Kotsiantis SB et al (2019). Multi-objective evolutionary optimization algorithms for machine learning: a recent survey. https://doi.org/10.1007/978-3-030-12767-1_4

  • Aszemi NM, Dominic P (2019) Hyperparameter optimization in convolutional neural network using genetic algorithms. Int J Adv Comput Sci Appl 10(6):269–278. https://doi.org/10.14569/IJACSA.2019.0100638

    Article  Google Scholar 

  • Baker B, Gupta O, Naik N et al (2017) Designing neural network architectures using reinforcement learning. In: Proceedings of the 2017 international conference on learning representations. https://doi.org/10.48550/arXiv.1611.02167

  • Bardenet R, Brendel M, Kégl B et al (2013) Collaborative hyperparameter tuning. In: Proceedings of the 2013 international conference on machine learning, pp 199–207. https://doi.org/10.5555/3042817.3042916

  • Bengio Y, Goodfellow I, Courville A (2017) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  • Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(2):281–305. https://doi.org/10.5555/2503308.2188395

  • Bergstra J, Bardenet R, Bengio Y et al (2011) Algorithms for hyper-parameter optimization. In: Proceedings of the 2011 annual conference on neural information processing systems, pp 2546–2554. https://doi.org/10.5555/2986459.2986743

  • Bochinski E, Senst T, Sikora T (2017) Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. In: Proceedings of the 2017 IEEE international conference on image processing, pp 3924–3928. https://doi.org/10.1109/ICIP.2017.8297018

  • Brock A, Lim T, Ritchie JM et al (2018) Smash: one-shot model architecture search through hypernetworks. In: Proceedings of the 2018 international conference on learning representation

  • Brockman G, Cheung V, Pettersson L et al (2016) Openai gym. arXiv:1606.01540

  • Carvalho P, Lourenço N, Assunção F, et al. (2020) Autolr: an evolutionary approach to learning rate policies. In: Proceedings of the 2020 genetic and evolutionary computation conference, pp 672–680. https://doi.org/10.1145/3377930.3390158

  • Chen Y, Meng G, Zhang Q et al (2019a) RENAS: reinforced evolutionary neural architecture search. In: Proceedings of the 2019 IEEE/CVF conference on computer vision and pattern recognition, pp 4782–4791. https://doi.org/10.1109/CVPR.2019.00492

  • Chen Z, Zhou Y, Huang Z (2019b) Auto-creation of effective neural network architecture by evolutionary algorithm and ResNet for image classification. In: Proceedings of the 2019 IEEE international conference on systems, man and cybernetics, Italy, pp 3895–3900. https://doi.org/10.1109/SMC.2019.8914267

  • Cheng AC, Lin CH, Juan DC et al (2020) Instanas: instance-aware neural architecture search. In: Proceedings of the AAAI conference on artificial intelligence, pp 3577–3584. https://doi.org/10.1609/aaai.v34i04.5764

  • Chung CC, Lin WT, Zhang R et al (2019) Emotion estimation by joint facial expression and speech tonality using evolutionary deep learning structures. In: Proceedings of the 2019 IEEE global conference on consumer electronics, pp 221–224. https://doi.org/10.1109/GCCE46687.2019.9015558

  • Cubuk ED, Zoph B, Man’e D et al (2019) Autoaugment: learning augmentation strategies from data. In: Proceedings of the 2019 IEEE/CVF Conference on computer vision and pattern recognition, pp 113–123. https://doi.org/10.1109/CVPR.2019.00020

  • Dauphin Y, Pascanu R, Gulcehre C et al (2014) Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In: Proceedings of the 2014 international conference on neural information processing systems, pp 2933–2941. https://doi.org/10.1083/jcb.111.6.2475

  • Deng J, Dong W, Socher R et al (2009) Imagenet: a large-scale hierarchical image database. In: Proceedings of the 2009 IEEE conference on computer vision and pattern recognition, pp 248–255. https://doi.org/10.1109/CVPR.2009.5206848

  • Ding S, Chen T, Gong X et al (2020) Autospeech: neural architecture search for speaker recognition. In: Proceedings of the 2020 international symposium on computing architecture. https://doi.org/10.21437/Interspeech.2020-1258

  • Dozat T (2016) Incorporating nesterov momentum into adam. In: Proceedings of the 2016 international conference on learning representations

  • Dufourq E, Bassett BA (2017a) Automated problem identification: regression vs classification via evolutionary deep networks. In: Proceedings of the 2017 South African Institute of computing sciences and information technology, pp 1–9. https://doi.org/10.1145/3129416.3129429

  • Dufourq E, Bassett BA (2017b) Eden: evolutionary deep networks for efficient machine learning. In: Proceedings of the 2017 international conference on pattern recognition association of South Africa and robotics and mechatronics, pp 110–115. https://doi.org/10.1109/RoboMech.2017.8261132

  • Eggensperger K, Feurer M, Hutter F et al (2013) Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In: Proceedings of the 2013 international conference on neural information processing systems, p 3. https://doi.org/10.1.1.737.6789

  • Eiben AE, Smith JE et al (2003) Introduction to evolutionary computing, vol 53. Springer, Berlin. https://doi.org/10.1007/978-3-662-05094-1

    Book  MATH  Google Scholar 

  • ElSaid A, Benson S, Patwardhan S et al (2019) Evolving recurrent neural networks for time series data prediction of coal plant parameters. In: Proceedings of the 2019 international conference on the applications of evolutionary computation, pp 488–503. https://doi.org/10.1007/978-3-030-16692-2_33

  • Elsken T, Metzen JH, Hutter F et al (2019) Neural architecture search: a survey. J Mach Learn Res 20(55):1–21

    MathSciNet  MATH  Google Scholar 

  • Esfahanian P, Akhavan M (2019) Gacnn: training deep convolutional neural networks with genetic algorithm. arXiv:1909.13354

  • Fan Z, Hu K, Li F et al (2016) Multiobjective evolutionary algorithms embedded with machine learning a survey. In: Proceedings of the 2016 IEEE congress on evolutionary computation, pp 1262–1266. https://doi.org/10.1109/CEC.2016.7743932

  • Fan Z, Wei J, Zhu G et al (2020) Evolutionary neural architecture search for retinal vessel segmentation. arXiv:2001.06678

  • Fedorov I, Adams RP, Mattina M et al (2019) Sparse: sparse architecture search for CNNs on resource-constrained microcontrollers. In: Proceedings of the 2019 conference on neural information processing systems, pp 4977–4989. https://doi.org/10.5555/3454287.3454735

  • Fieldsend J, Singh S (2005) Pareto evolutionary neural networks. IEEE Trans Neural Netw 16(2):338–354. https://doi.org/10.1109/TNN.2004.841794

    Article  Google Scholar 

  • Galvan E, Mooney P (2021) Neuroevolution in deep neural networks: current trends and future challenges. IEEE Trans Artif Intell. https://doi.org/10.1109/TAI.2021.3067574

    Article  Google Scholar 

  • Gao Z, Pan Z, Zuo C et al (2019) An optimized deep network representation of multimutation differential evolution and its application in seismic inversion. IEEE Trans Geosci Remote Sens 57(7):4720–4734. https://doi.org/10.1109/TGRS.2019.2892567

    Article  Google Scholar 

  • Gao Z, Li Y, Yang Y et al (2020) A GPSO-optimized convolutional neural networks for EEG-based emotion recognition. Neurocomputing 380:225–235. https://doi.org/10.1016/j.neucom.2019.10.096

    Article  Google Scholar 

  • Ghiasi G, Lin TY, Le QV (2019) Nas-fpn: Learning scalable feature pyramid architecture for object detection. In: Proceedings of the 2019 IEEE/CVF conference on computer vision and pattern recognition, pp 7029–7038. https://doi.org/10.1109/CVPR.2019.00720

  • Golovin D, Solnik B, Moitra S et al (2017) Google vizier: a service for black-box optimization. In: Proceedings of the 2017 ACM SIGKDD conference on knowledge discovery and data mining, pp 1487–1495. https://doi.org/10.1145/3097983.3098043

  • Gong X, Chang S, Jiang Y et al (2019a) Autogan: neural architecture search for generative adversarial networks. In: Proceedings of the 2019 IEEE/CVF international conference on computer vision, pp 3223–3233. https://doi.org/10.1109/ICCV.2019.00332

  • Gong X, Chang S, Jiang Y et al (2019b) Autogan: neural architecture search for generative adversarial networks. In: Proceedings of the 2019 IEEE/CVF international conference on computer vision, Korea, pp 3224–3234. https://doi.org/10.1109/ICCV.2019.00332

  • Goodfellow IJ, Bulatov Y, Ibarz J et al (2013) Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv:1312.6082

  • Goodfellow I, Vinyals O, Saxe A (2015) Qualitatively characterizing neural network optimization problems. In: Proceedings of the 2015 international conference on learning representations. https://doi.org/10.48550/arXiv.1412.6544

  • Guo Z, Zhang X, Mu H et al (2020) Single path one-shot neural architecture search with uniform sampling. In: Proceedings of the 2020 European conference on computer vision, pp 544–560. https://doi.org/10.1007/978-3-030-58517-4_32

  • Han JH, Choi DJ, Park SU et al (2020) Hyperparameter optimization using a genetic algorithm considering verification time in a convolutional neural network. J Electr Eng Technol 15(2):721–726. https://doi.org/10.1007/s42835-020-00343-7

    Article  Google Scholar 

  • He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, USA, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  • He X, Zhao K, Chu X (2021) Automl: a survey of the state-of-the-art. Knowl-Based Syst 212(106):622. https://doi.org/10.1016/j.knosys.2020.106622

    Article  Google Scholar 

  • Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507. https://doi.org/10.1126/science.1127647

    Article  MathSciNet  MATH  Google Scholar 

  • Hossain D, Capi G (2018) Multiobjective evolution of deep learning parameters for robot manipulator object recognition and grasping. Adv Robot 32(20):1090–1101. https://doi.org/10.1016/j.procs.2017.01.195

    Article  Google Scholar 

  • Hu S, Cheng R, He C et al (2021) Multi-objective neural architecture search with almost no training. Evolut MultiCriterion Optim 12654:492–503. https://doi.org/10.1007/978-3-030-72062-9_39

    Article  MathSciNet  Google Scholar 

  • Huang G, Liu Z, Van Der Maaten L et al (2017) Densely connected convolutional networks. In: Proceedings of the 2017 IEEE conference on computer vision and pattern recognition, Hawaii, pp 4700–4708. https://doi.org/10.1109/CVPR.2017.243

  • Hutter F, Hoos HH, Leyton-Brown K (2011) Sequential model-based optimization for general algorithm configuration. In: Proceedings of the 2011 learning and intelligence optimization, pp 507–523. https://doi.org/10.1007/978-3-642-25566-3_40

  • Itano F, de Sousa MAdA, Del-Moral-Hernandez E (2018) Extending MLP ANN hyper-parameters optimization by using genetic algorithm. In: Proceedings of the 2018 International Joint Conference on Neural Network, pp 1–8. https://doi.org/10.1109/IJCNN.2018.8489520

  • Jaddi NS, Abdullah S, Hamdan AR (2016) A solution representation of genetic algorithm for neural network weights and structure. Inf Process Lett 116(1):22–25. https://doi.org/10.13140/RG.2.1.5178.2888

    Article  Google Scholar 

  • Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21(4):345–383. https://doi.org/10.1023/A:1012771025575

    Article  MathSciNet  MATH  Google Scholar 

  • Junior FEF, Yen GG (2019) Particle swarm optimization of deep neural networks architectures for image classification. Swarm Evol Comput 49:62–74. https://doi.org/10.1016/j.swevo.2019.05.010

    Article  Google Scholar 

  • Kanada Y (2016) Optimizing neural-network learning rate by using a genetic algorithm with per-epoch mutations. In: Proceedings of the 2016 International Joint Conference on Neural Network, pp 1472–1479. https://doi.org/10.1109/IJCNN.2016.7727372

  • Klein A, Falkner S, Bartels S et al (2017) Fast Bayesian optimization of machine learning hyperparameters on large datasets. In: Proceedings of the 2017 international conference on artificial intelligence and statistics, pp 528–536

  • Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Handb Syst Autoimmune Dis 1(4):1–60. https://doi.org/10.1.1.222.9220

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25(6):1097–1105. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  • Lecun Y, Bottou L, Bengio Y et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  • Li L, Jamieson KG, DeSalvo G et al (2017) Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In: Proceedings of the 2017 international conference on learning representations. https://doi.org/10.5555/3122009.3242042

  • Li X, Zhou Y, Pan Z et al (2019) Partial order pruning: for best speed/accuracy trade-off in neural architecture search. In: Proc. of the IEEE/CVF conference on computer vision and pattern recognition, pp 9137–9145. https://doi.org/10.1109/CVPR.2019.00936

  • Liang J, Meyerson E, Hodjat B et al (2019) Evolutionary neural autoML for deep learning. In: Proceedings of the 2019 genetic and evolutionary computation conference, pp 401–409. https://doi.org/10.1145/3321707.3321721

  • Liu C, Zoph B, Neumann M et al (2018a) Progressive neural architecture search. In: Proceedings of the 2018 European conference on computer vision, pp 19–34. https://doi.org/10.1007/978-3-030-01246-5_2

  • Liu H, Simonyan K, Vinyals O et al (2018b) Hierarchical representations for efficient architecture search. In: Proceedings of the 2018 international conference on learning representations

  • Liu H, Simonyan K, Yang Y (2019) Darts: differentiable architecture search. In: Proceedings of the 2019 international conference on learning representations

  • Liu Y, Sun Y, Xue B et al (2021) A survey on evolutionary neural architecture search. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3100554

    Article  Google Scholar 

  • Lorenzo PR, Nalepa J (2018) Memetic evolution of deep neural networks. In: Proceedings of the 2018 genetic and evolutionary computation conference, pp 505–512. https://doi.org/10.1145/3205455.3205631

  • Lorenzo PR, Nalepa J, Kawulok M et al (2017) Particle swarm optimization for hyper-parameter selection in deep neural networks. In: Proceedings of the 2017 genetic and evolutionary computation conference, pp 481–488. https://doi.org/10.1145/3071178.3071208

  • Loshchilov I, Hutter F (2016) CMA-ES for hyperparameter optimization of deep neural networks. arXiv:1604.07269

  • Lu Z, Whalen I, Boddeti V et al (2019) Nsga-net: neural architecture search using multi-objective genetic algorithm. In: Proceedings of the 2019 genetic and evolutionary computation conference, pp 419–427. https://doi.org/10.1145/3321707.3321729

  • Majidi M, Toroghi RM (2021) Music harmony generation, through deep learning and using a multi-objective evolutionary algorithm. arXiv:2102.07960

  • Martín A, Lara-Cabrera R, Fuentes-Hurtado F et al (2018) Evodeep: a new evolutionary approach for automatic deep neural networks parametrisation. J Parallel Distrib Comput 117:180–191. https://doi.org/10.1016/j.jpdc.2017.09.006

    Article  Google Scholar 

  • Miller GF, Todd PM, Hegde SU (1989) Designing neural networks using genetic algorithms. In: Proceedings of the 1989 international conference on genetic algorithms, pp 379–384. https://doi.org/10.5555/645512.657097

  • Montana DJ, Davis L (1989) Training feedforward neural networks using genetic algorithms. In: Proceedings of the 1989 international joint conference on artificial intelligence, pp 762–767. https://doi.org/10.5555/1623755.1623876

  • Negrinho R, Gordon G (2017) Deeparchitect: automatically designing and training deep architectures. arXiv:1704.08792

  • Oliveira P, Portela C, Santos M et al (2015) Machine learning: an overview of optimization techniques. Recent Adv Comput Sci 32:51–56

    Google Scholar 

  • Palmes P, Hayasaka T, Usui S (2005) Mutation-based genetic neural network. IEEE Trans Neural Netw 16(3):587–600. https://doi.org/10.1109/TNN.2005.844858

    Article  Google Scholar 

  • Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: Proceedings of the 2013 international conference on machine learning, pp 1310–1318. https://doi.org/10.1007/s12088-011-0245-8

  • Pearlmutter B (1991) Gradient descent: second order momentum and saturating error. Adv Neural Inf Process Syst 4:887–894. https://doi.org/10.5555/2986916.2987025

    Article  Google Scholar 

  • Peng J, Sun M, Zhang Z et al (2019) Efficient neural architecture transformation search in channel-level for object detection. arXiv:1909.02293

  • Pham H, Guan M, Zoph B et al (2018) Efficient neural architecture search via parameters sharing. In: Proceedings of the 2018 international conference on machine learning, pp 4095–4104

  • Probst P, Wright MN, Boulesteix AL (2019) Hyperparameters and tuning strategies for random forest. Wiley Interdiscip Rev Data Min and Knowl Discov 9(3):e1301. https://doi.org/10.1002/widm.1301

    Article  Google Scholar 

  • Rasmussen CE (2003) Gaussian processes in machine learning. In: Summer school on machine learning, pp 63–71. https://doi.org/10.1007/978-3-540-28650-9_4

  • Real E, Moore S, Selle A et al (2017) Large-scale evolution of image classifiers. In: Proceedings of the 2017 international conference on machine learning, pp 2902–2911. https://doi.org/10.5555/3305890.3305981

  • Real E, Aggarwal A, Huang Y et al (2019) Regularized evolution for image classifier architecture search. In: Proceedings of the 2019 AAAI conference on artificial intelligence, pp 4780–4789. https://doi.org/10.1609/aaai.v33i01.33014780

  • Ren P, Xiao Y, Chang X et al (2021) A comprehensive survey of neural architecture search: challenges and solutions. ACM Comput Surv 54:1–34. https://doi.org/10.1145/3447582

    Article  Google Scholar 

  • Risi S, Togelius J (2017) Neuroevolution in games: state of the art and open challenges. IEEE Trans Comput Intell AI Games 9(1):25–41. https://doi.org/10.1109/TCIAIG.2015.2494596

    Article  Google Scholar 

  • Rodriguez J (2018) Understanding hyperparameters optimization in deep learning models: concepts and tools

  • Rojas R (2013) Neural networks: a systematic introduction. Springer

  • Sabara MA, Somantri O, Nurcahyo H et al (2019) Diagnosis classification of dengue fever based on neural networks and genetic algorithms. J Phys Conf Ser 1175(1):012065. https://doi.org/10.1088/1742-6596/1175/1/012065

    Article  Google Scholar 

  • Sanders S, Giraud Carrier C (2017) Informing the use of hyperparameter optimization through metalearning. In: Proceedings of the 2017 IEEE international conference on data mining, pp 1051–1056. https://doi.org/10.1109/ICDM.2017.137

  • Schrijver A (2005) On the history of combinatorial optimization (till 1960). Handb Oper Res Manag Sci 12:1–68. https://doi.org/10.1016/S0927-0507(05)12001-5

  • Shinozaki T, Watanabe S (2015) Structure discovery of deep neural network based on evolutionary algorithms. In: Proceedings of the 2015 IEEE international conference on acoustics, speech and signal processing, pp 4979–4983.https://doi.org/10.1109/ICASSP.2015.7178918

  • Silver D, Huang A, Maddison CJ et al (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489. https://doi.org/10.1038/nature16961

    Article  Google Scholar 

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  • Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. In: Proceedings of the 2012 international conference on neural information processing systems, pp 2951–2959.https://doi.org/10.5555/2999325.2999464

  • Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127. https://doi.org/10.1162/106365602320169811

    Article  Google Scholar 

  • Stanley KO, Clune J, Lehman J et al (2019) Designing neural networks through neuroevolution. Nat Mach Intell 1(1):24–35. https://doi.org/10.1038/s42256-018-0006-z

    Article  Google Scholar 

  • Stoica P, Gershman A (1999) Maximum-likelihood DOA estimation by data-supported grid search. IEEE Signal Proc Lett 6(10):273–275. https://doi.org/10.1109/97.789608

    Article  Google Scholar 

  • Such FP, Madhavan V, Conti E et al (2017) Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv:1712.06567

  • Suganuma M, Shirakawa S, Nagao T (2020) Designing convolutional neural network architectures using cartesian genetic programming. In: Proceedings of the 2020 deep neural evolution, pp 185–208. https://doi.org/10.1007/978-981-15-3685-4_7

  • Sun Y, Xue B, Zhang M et al (2019) Completely automated CNN architecture design based on blocks. IEEE Trans Neural Netw Learn Syst 31(4):1242–1254. https://doi.org/10.1109/TNNLS.2019.2919608

    Article  MathSciNet  Google Scholar 

  • Sun Y, Xue B, Zhang M et al (2020) Evolving deep convolutional neural networks for image classification. IEEE Trans Evol Comput 24(2):394–407. https://doi.org/10.1109/TEVC.2019.2916183

    Article  Google Scholar 

  • Sutton R (1986) Two problems with back propagation and other steepest descent learning procedures for networks. In: Proceedings of the 1986 annual conference of the cognitive science society, pp 823–832

  • Szegedy C, Liu W, Jia Y et al (2015) Going deeper with convolutions. In: Proceedings of the 2015 IEEE international conference on computer vision and pattern recognition, pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594

  • Tanaka T, Moriya T, Shinozaki T et al (2016a) Automated structure discovery and parameter tuning of neural network language model based on evolution strategy. In: Proceedings of the 2016 IEEE spoken language technology workshop, pp 665–671. https://doi.org/10.1109/SLT.2016.7846334

  • Tanaka T, Shinozaki T, Watanabe S et al (2016b) Evolution strategy based neural network optimization and LSTM language model for robust speech recognition. In: Proceedings of the 2016 international workshop on speech processing in everyday environment, pp 32–35

  • Tang X, Wang H, Jiao C (2019) Multi-objective evolutionary metric learning for image retrieval using convolutional neural network features. In: Proceedings of the 2019 IEEE congress on evolutionary computation, pp 578–585. https://doi.org/10.1109/CEC.2019.8790231

  • Tirumala SS (2014) Implementation of evolutionary algorithms for deep architectures. In: Proceedings of the 2014 international workshop on artificial intelligence and cognition, pp 164–171

  • Turner AJ, Miller JF (2014) Neuroevolution: evolving heterogeneous artificial neural networks. Evol Intell 7(3):135–154. https://doi.org/10.1007/s12065-014-0115-5

    Article  Google Scholar 

  • Verma M, Sinha P, Goyal K et al (2019) A novel framework for neural architecture search in the hill climbing domain. In: Proceedings of the 2019 IEEE international conference on artificial intelligence and knowledge engineering, pp 1–8. https://doi.org/10.1109/AIKE.2019.00009

  • Volna E (2010) Neuroevolutionary optimization. Int J Comput Sci Issues 7(4). https://doi.org/10.48550/arXiv.1004.3557

  • Wang Y, Zhang H, Zhang G (2019) CPSO-CNN: an efficient PSO-based algorithm for fine-tuning hyper-parameters of convolutional neural networks. Swarm Evol Comput 49:114–123. https://doi.org/10.1016/j.swevo.2019.06.002

    Article  Google Scholar 

  • Wei T, Wang C, Rui Y et al (2016) Network morphism. In: Proceedings of the 2016 international conference on machine learning, pp 564–572. https://doi.org/10.5555/3045390.3045451

  • Wen W, Liu H, Chen Y et al (2020) Neural predictor for neural architecture search. In: Proceedings of the 2020 European conference on computer vision, pp 660–676. https://doi.org/10.1007/978-3-030-58526-6_39

  • Whiteley D (1988) Applying genetic algorithms to neural network problems. Neural Netw 1:230. https://doi.org/10.1016/0893-6080(88)90267-5

    Article  Google Scholar 

  • Wierstra D, Schaul T, Peters J et al (2008) Natural evolution strategies. In: Proceedings of the 2008 IEEE congress on evolutionary computation, pp 3381–3387. https://doi.org/10.5555/2627435.2638566

  • Wistuba M, Rawat A, Pedapati T (2019) A survey on neural architecture search. J Mach Learn Res 20:1–21

    Google Scholar 

  • Wu Y, Mansimov E, Liao S et al (2017) Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation. In: Proceedings of the 2017 conference on neural information processing systems, pp 5285–5294. https://doi.org/10.5555/3295222.3295280

  • Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747

  • Xiao X, Yan M, Basodi S, et al. (2020) Efficient hyperparameter optimization in deep learning using a variable length genetic algorithm. arXiv:2006.12703

  • Xie L, Yuille A (2017) Genetic CNN. In: Proceedings of the 2017 IEEE international conference on computer vision, pp 1379–1388. https://doi.org/10.1109/ICCV.2017.154

  • Xie S, Zheng H, Liu C et al (2019) SNAS: stochastic neural architecture search. In: Proceedings of the 2019 international conference on learning representations

  • Xu B, Wang N, Chen T et al (2015) Empirical evaluation of rectified activations in convolutional network. arXiv:1505.00853

  • Xue B, Zhang M, Browne WN et al (2016) A survey on evolutionary computation approaches to feature selection. IEEE Trans Evol Comput 20(4):606–626. https://doi.org/10.1109/TEVC.2015.2504420

    Article  Google Scholar 

  • Yao X (1999) Evolving artificial neural networks. Proc IEEE 87(9):1423–1447. https://doi.org/10.1109/5.784219

    Article  Google Scholar 

  • Ye Q, Han Y, Sun Y, et al. (2020) PSO-PS: parameter synchronization with particle swarm optimization for distributed training of deep neural networks. In: Proceedings of the 2020 international joint conference on neural network, pp 1–8. https://doi.org/10.1109/IJCNN48605.2020.9207698

  • Ying C, Klein A, Christiansen E et al (2019) Nas-bench-101: towards reproducible neural architecture search. In: Proceedings of the 2019 international conference on machine learning, pp 7105–7114. https://doi.org/10.1007/978-3-030-05318-5_3

  • Yu T, Zhu H (2020) Hyper-parameter optimization: a review of algorithms and applications

  • Zhang R, Tao J (2017) A nonlinear fuzzy neural network modeling approach using an improved genetic algorithm. IEEE Trans Ind Electron 65(7):5882–5892. https://doi.org/10.1109/TIE.2017.2777415

    Article  Google Scholar 

  • Zhang J, Zhan ZH, Lin Y et al (2011) Evolutionary computation meets machine learning: a survey. Comput Intell Mag 6:68–75. https://doi.org/10.1109/MCI.2011.942584

    Article  Google Scholar 

  • Zhang C, Lim P, Qin AK et al (2016) Multiobjective deep belief networks ensemble for remaining useful life estimation in prognostics. IEEE Trans Neural Netw Learn Syst 28(10):2306–2318. https://doi.org/10.1109/TNNLS.2016.2582798

    Article  Google Scholar 

  • Zhang H, Kiranyaz S, Gabbouj M (2018) Finding better topologies for deep convolutional neural networks by evolution. arXiv:1809.03242

  • Zhang Y, Qiu Z, Liu J et al (2019) Customizable architecture search for semantic segmentation. In: Proceedings of the 2019 IEEE/CVF conference on computer vision and pattern recognition, pp 11633–11642. https://doi.org/10.1109/CVPR.2019.01191

  • Zhang X, Huang Z, Wang N et al (2020) You only search once: single shot neural architecture search via direct sparse optimization. IEEE Trans Pattern Anal Mach Intell 43(9):1–16. https://doi.org/10.1109/TPAMI.2020.3020300

    Article  Google Scholar 

  • Zhong Z, Yang Z, Deng B et al (2020) Blockqnn: efficient block-wise neural network architecture generation. IEEE Trans Pattern Anal Mach Intell 43(7):1. https://doi.org/10.1109/TPAMI.2020.2969193

    Article  Google Scholar 

  • Zoller MA, Huber MF (2021) Benchmark and survey of automated machine learning frameworks. J Artif Intell Res 70:409–472. https://doi.org/10.1613/jair.1.11854

    Article  MathSciNet  MATH  Google Scholar 

  • Zoph B, Le QV (2017) Neural architecture search with reinforcement learning. In: Proceedings of the 2017 international conference on learning representation

  • Zoph B, Vasudevan V, Shlens J et al (2018) Learning transferable architectures for scalable image recognition. In: Proceedings of the 2018 IEEE conference on computer vision and pattern recognition, pp 8697–8710. https://doi.org/10.1109/CVPR.2018.00907

Download references

Funding

This work was supported in part by the Key Project of Science and Technology Innovation 2030 supported by the Ministry of Science and Technology of China under Grant 2018AAA0101302 and in part by the General Program of National Natural Science Foundation of China (NSFC) under Grant 61773300.

Author information

Authors and Affiliations

  1. Guangzhou Institute of Technology, Xidian University, Guangzhou, 510555, Guangdong, China

    Yifan Li & Jing Liu

Authors
  1. Yifan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. YL was responsible for material preparation and the literature search and wrote the first draft of the manuscript. JL reviewed and edited the previous versions of the manuscript and provided funding and supervision.

Corresponding author

Correspondence to Jing Liu.

Ethics declarations

Conflict of interest

The authors have no relevant financial or nonfinancial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Liu, J. A survey: evolutionary deep learning. Soft Comput 27, 9401–9423 (2023). https://doi.org/10.1007/s00500-023-08316-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-08316-4

Keywords

Search

Navigation