Skip to main content
SpringerLink
Log in

Evolutionary Neural Network Architecture Search

  • Chapter
  • First Online:
Handbook of Evolutionary Machine Learning

Part of the book series: Genetic and Evolutionary Computation ((GEVO))

  • 1446 Accesses

Abstract

Deep Neural Networks (DNNs) have been remarkably successful in numerous scenarios of machine learning. However, the typical design for DNN architectures is manual, which highly relies on the domain knowledge and experience of neural networks. Neural architecture search (NAS) methods are often considered an effective way to achieve automated design of DNN architectures. There are three approaches to realizing NAS: reinforcement learning approaches, gradient-based approaches, and evolutionary computation approaches. Among them, evolutionary computation-based NAS (ENAS) has received much attention. This chapter will detail ENAS in terms of four aspects. First, we will present an overall introduction to NAS and the commonly used approaches to NAS. Following that, we will introduce the core components of ENAS and discuss the details of how to design an ENAS algorithm with a focus on search space, search strategy, and performance evaluation of the ENAS algorithm. Moreover, detailed implementations of these components will be presented to help readers implement an ENAS algorithm step by step. We will discuss state-of-the-art ENAS methods with the three core components. Finally, we will provide five major challenges and identify corresponding future directions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Assunção, F., Correia, J., Conceição, R., Pimenta, M.J.M., Tomé, B., Lourenço, N., Machado, P.: Automatic design of artificial neural networks for gamma-ray detection. IEEE Access 7, 110531–110540 (2019)

    Article  Google Scholar 

  2. Bäck, T., Fogel, D.B., Michalewicz, Z.: Handbook of evolutionary computation. Release 97(1), B1 (1997)

    MATH  Google Scholar 

  3. Bender, G., Kindermans, P.J., Zoph, B., Vasudevan, V., Le, Q.: Understanding and simplifying one-shot architecture search. In: International Conference on Machine Learning, pp. 550–559. PMLR (2018)

    Google Scholar 

  4. Bi, Y., Xue, B., Zhang, M.: An evolutionary deep learning approach using genetic programming with convolution operators for image classification. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp. 3197–3204. IEEE (2019)

    Google Scholar 

  5. Broni-Bediako, C., Murata, Y., Mormille, L.H., Atsumi, M.: Evolutionary NAS with gene expression programming of cellular encoding. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 2670–2676. IEEE (2020)

    Google Scholar 

  6. Chen, Z., Zhou, Y., Huang, Z.: Auto-creation of effective neural network architecture by evolutionary algorithm and resnet for image classification. In: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 3895–3900. IEEE2019

    Google Scholar 

  7. Cheng, F., Jiangsheng, Yu., Xiong, H.: Facial expression recognition in Jaffe dataset based on Gaussian process classification. IEEE Trans. Neural Netw. 21(10), 1685–1690 (2010)

    Article  Google Scholar 

  8. Deng, B., Yan, J., Lin, D.: Peephole: Predicting network performance before training (2017). arXiv:1712.03351

  9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding (2018). arXiv:1810.04805

  10. Doerr, B., Doerr, C.: Theory of parameter control for discrete black-box optimization: Provable performance gains through dynamic parameter choices. Theory of Evolutionary Computation, pp. 271–321 (2020)

    Google Scholar 

  11. Doerr, B., Happ, E., Klein, C.: Crossover can provably be useful in evolutionary computation. In: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, pp. 539–546 (2008)

    Google Scholar 

  12. Dong, X., Yang, Y.: Nas-bench-201: extending the scope of reproducible neural architecture search. In: International Conference on Learning Representations (2019)

    Google Scholar 

  13. Elsken, T., Metzen, J.H., Hutter, F.: Simple and efficient architecture search for convolutional neural networks (2017). arXiv:1711.04528

  14. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(1), 1997–2017 (2019)

    Google Scholar 

  15. Evans, B., Al-Sahaf, H., Xue, B., Zhang, M.: Evolutionary deep learning: a genetic programming approach to image classification. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–6. IEEE (2018)

    Google Scholar 

  16. Evans, B.P., Al-Sahaf, H., Xue, B., Zhang, M.: Genetic programming and gradient descent: A memetic approach to binary image classification (2019). arXiv:1909.13030

  17. Fan, Z., Wei, J., Zhu, G., Mo, J., Li, W.: Evolutionary neural architecture search for retinal vessel segmentation (2020). arXiv:2001.06678

  18. Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: from architectures to learning. Evol. Intel. 1(1), 47–62 (2008)

    Article  Google Scholar 

  19. Fujino, S., Naoki, M., Matsumoto, K.: Deep convolutional networks for human sketches by means of the evolutionary deep learning. In: 2017 Joint 17th World Congress of International Fuzzy Systems Association and 9th International Conference on Soft Computing and Intelligent Systems (IFSA-SCIS), pp. 1–5. IEEE (2017)

    Google Scholar 

  20. Gunantara, N.: A review of multi-objective optimization: methods and its applications. Cogent Eng. 5(1), 1502242 (2018)

    Article  Google Scholar 

  21. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  22. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications (2017). arXiv:1704.04861

  23. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  24. Irwin-Harris, W., Sun, Y., Xue, B., Zhang, M.: A graph-based encoding for evolutionary convolutional neural network architecture design. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp. 546–553. IEEE (2019)

    Google Scholar 

  25. Johner, F.M., Wassner, J.: Efficient evolutionary architecture search for CNN optimization on gtsrb. In: 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 56–61. IEEE (2019)

    Google Scholar 

  26. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)

    Google Scholar 

  27. Koza, J.R.: Genetic programming as a means for programming computers by natural selection. Stat. Comput. 4(2), 87–112 (1994)

    Google Scholar 

  28. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Handbook Syst. Autoimmune Dis. 1(4) (2009)

    Google Scholar 

  29. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Google Scholar 

  30. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  31. Li, S., Sun, Y., Yen, G.G., Zhang, M.: Automatic design of convolutional neural network architectures under resource constraints, IEEE Trans. Neural Netw. Learn. Syst. (2021)

    Google Scholar 

  32. Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search (2018). arXiv:1806.09055

  33. Liu, J., Gong, M., Miao, Q., Wang, X., Li, H.: Structure learning for deep neural networks based on multiobjective optimization. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2450–2463 (2017)

    Article  MathSciNet  Google Scholar 

  34. Liu, P., El Basha, M.D., Li, Y., Xiao, Y., Sanelli, P.C., Fang, R.: Deep evolutionary networks with expedited genetic algorithms for medical image denoising. Med. Image Anal. 54, 306–315 (2019)

    Google Scholar 

  35. Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G.G., Tan, K.C.: A survey on evolutionary neural architecture search, IEEE Trans. Neural Netw. Learn. Syst. (2021)

    Google Scholar 

  36. Loni, M., Sinaei, S., Zoljodi, A., Daneshtalab, M., Sjödin, M.: Deepmaker: a multi-objective optimization framework for deep neural networks in embedded systems. Microprocess. Microsyst. 73, 102989 (2020)

    Article  Google Scholar 

  37. Lu, Z., Whalen, I., Boddeti, V., Dhebar, Y., Deb, K., Goodman, E., Banzhaf, W.: NSGA-Net: neural architecture search using multi-objective genetic algorithm. In: Proceedings of the Genetic and Evolutionary Computation conference, pp. 419–427 (2019)

    Google Scholar 

  38. Lu, Z., Whalen, I., Dhebar, Y., Deb, K., Goodman, E.D., Banzhaf, W., Boddeti, V.N.: Multiobjective evolutionary design of deep convolutional neural networks for image classification. IEEE Trans. Evol. Comput. 25(2), 277–291 (2020)

    Google Scholar 

  39. Rawal, A., Miikkulainen, R.: From nodes to networks: evolving recurrent neural networks (2018). arXiv:1803.04439

  40. Real, E., Moore, S., Selle, A., Saxena, S., Suematsu, Y.L., Tan, J., Le, Q.V., Kurakin, A.: Large-scale evolution of image classifiers. In: International Conference on Machine Learning, pp. 2902–2911. PMLR (2017)

    Google Scholar 

  41. Ruder, S.: An overview of gradient descent optimization algorithms (2016). arXiv:1609.04747

  42. Russakovsky, O., Deng, J., Hao, S., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  43. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv:1409.1556

  44. So, D., Le, Q., Liang, C.: The evolved transformer. In: International Conference on Machine Learning, pp. 5877–5886. PMLR (2019)

    Google Scholar 

  45. Song, D., Chang, X., Jia, X., Chen, Y., Chunjing, X., Wang, Y.: Efficient residual dense block search for image super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12007–12014 (2020)

    Google Scholar 

  46. Stanley, K.O., Bryant, B.D., Miikkulainen, R.: Real-time neuroevolution in the NERO video game. IEEE Trans. Evol. Comput. 9(6), 653–668 (2005)

    Google Scholar 

  47. Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

    Google Scholar 

  48. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Google Scholar 

  49. Sun, Y., Xue, B., Zhang, M., Yen, G.G.: A particle swarm optimization-based flexible convolutional autoencoder for image classification. IEEE Trans. Neural Netw. Learn. Syst. 30(8), 2295–2309 (2018)

    Google Scholar 

  50. Sun, Y., Xue, B., Zhang, M., Yen, G.G.: Completely automated CNN architecture design based on blocks. IEEE Trans. Neural Netw. Learn. Syst. 31(4), 1242–1254 (2019)

    Google Scholar 

  51. Sun, Y., Xue, B., Zhang, M., Yen, G.G.: Evolving deep convolutional neural networks for image classification. IEEE Trans. Evol. Comput. 24(2), 394–407 (2019)

    Google Scholar 

  52. Sun, Y., Xue, B., Zhang, M., Yen, G.G., Lv, J.: Automatically designing CNN architectures using the genetic algorithm for image classification. IEEE Trans. Cybern. 50(9), 3840–3854 (2020)

    Google Scholar 

  53. Sun, Y., Yen, G.G., Zhang, M.: Internet protocol based architecture design. In: Evolutionary Deep Neural Architecture Search: Fundamentals. Methods, and Recent Advances, pp. 181–192. Springer, Cham (2023)

    Google Scholar 

  54. Sze, V., Chen, Y.H., Yang, T.J., Emer, J.S.: Efficient processing of deep neural networks: a tutorial and survey. Proc. IEEE 105(12), 2295–2329 (2017)

    Google Scholar 

  55. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  56. Wang, B., Sun, Y., Xue, B., Zhang, M.: Evolving deep convolutional neural networks by variable-length particle swarm optimization for image classification. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2018)

    Google Scholar 

  57. Wang, B., Xue, B., Zhang, M.: Particle swarm optimisation for evolving deep neural networks for image classification by evolving and stacking transferable blocks. In: 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2020)

    Google Scholar 

  58. Wu, T., Shi, J., Zhou, D., Lei, Y., Gong, M.: A multi-objective particle swarm optimization for neural networks pruning. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp. 570–577. IEEE (2019)

    Google Scholar 

  59. Xie, L., Yuille, A.: Genetic CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1379–1388 (2017)

    Google Scholar 

  60. Xie, X., Liu, Y., Sun, Y., Yen, G.G., Xue, B., Zhang, M.: Benchenas: a benchmarking platform for evolutionary neural architecture search. IEEE Trans. Evol. Comput. (2022)

    Google Scholar 

  61. Yang, Z., Wang, Y., Chen, X., Shi, B., Xu, C., Xu, C., Tian, Q., Xu, C.: Cars: continuous evolution for efficient neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1829–1838 (2020)

    Google Scholar 

  62. Yao, X.: Evolving artificial neural networks. Proc. IEEE 87(9), 1423–1447 (1999)

    Article  Google Scholar 

  63. Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: Nas-bench-101: towards reproducible neural architecture search. In: International Conference on Machine Learning, pp. 7105–7114. PMLR (2019)

    Google Scholar 

  64. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision, pp. 818–833. Springer (2014)

    Google Scholar 

  65. Zhang, M.: Evolutionary deep learning for image analysis. Talk at World Congress on Computational Intelligence (WCCI) (2018). Published on July 2, 2020

    Google Scholar 

  66. Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)

    Google Scholar 

  67. Zhou, Z.-H., Yang, Yu., Qian, C.: Evolutionary Learning: Advances in Theories and Algorithms. Springer, Singapore (2019)

    Book  MATH  Google Scholar 

  68. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning (2016). arXiv:1611.01578

  69. Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, 610064, China

    Zeqiong Lv, Xiaotian Song, Yuqi Feng, Yuwei Ou & Yanan Sun

  2. Centre for Data Science and Artificial Intelligence and School of Engineering and Computer Science, Victoria University of Wellington, Wellington, 6140, New Zealand

    Mengjie Zhang

Authors
  1. Zeqiong Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaotian Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuqi Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuwei Ou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mengjie Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanan Sun .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, USA

    Wolfgang Banzhaf

  2. Department of Informatics Engineering, University of Coimbra, Coimbra, Portugal

    Penousal Machado

  3. School of Engineering and Computer Science and Centre for Data Science and Artificial Intelligence, Victoria University of Wellington, Wellington, New Zealand

    Mengjie Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Lv, Z., Song, X., Feng, Y., Ou, Y., Sun, Y., Zhang, M. (2024). Evolutionary Neural Network Architecture Search. In: Banzhaf, W., Machado, P., Zhang, M. (eds) Handbook of Evolutionary Machine Learning. Genetic and Evolutionary Computation. Springer, Singapore. https://doi.org/10.1007/978-981-99-3814-8_9

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-3814-8_9

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-3813-1

  • Online ISBN: 978-981-99-3814-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation