Abstract
Neural evolutionary computation has risen as a promising approach to propose neural network architectures without human interference. However, the often high computational cost of these approaches is a serious challenge for their application and research. In this work, we empirically analyse standard practices with Coevolution of Deep NeuroEvolution of Augmenting Topologies (CoDeepNEAT) and the effect that different initialization functions have when experiments are tuned for quick evolving networks on a small number of generations and small populations. We compare networks initialized with the He, Glorot, and Random initializations on different settings of population size, number of generations, training epochs, etc. Our results suggest that properly setting hyperparameters for short training sessions in each generation may be sufficient to produce competitive neural networks. We also observed that the He initialization, when associated with neural evolution, has a tendency to create architectures with multiple residual connections, while the Glorot initializer has the opposite effect.
We thank Coordination for the Improvement of Higher Education Personnel - CAPES/PROAP and Amazonas State Research Support Foundation - FAPEAM/POSGRAD 2021. This research was partially supported by CAPES via student support grant #88887.498437/2020-00.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ma, Y., Xie, Y.: Evolutionary neural networks for deep learning: a review. Int. J. Mach. Learn. Cybern. (2022). https://doi.org/10.1007/s13042-022-01578-8
Kumar, S. K.: On weight initialization in deep neural networks. In: arXiv preprint arXiv:1704.08863 (2017)
Initializing neural networks. https://www.deeplearning.ai/ai-notes/initialization/. Accessed 12 June 2022
Goodfellow, I.J., Bengio, Y., Courville, A.: Deep Learning, 1st edn. Cambridge (2016)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256. JMLR Workshop and Conference Proceedings, Sardinia (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034. IEEE, Santiago (2010)
Koutnik, J., Gomez, F., Schmidhuber, J.: Evolving neural networks in compressed weight space. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, pp. 619–626. ACM, Portland (2010)
Togelius, J., Gomez, F., Schmidhuber, J.: Learning what to ignore: memetic climbing in topology and weight space. In: Proceedings of the 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 3274–3281. IEEE, Hong Kong (2008)
Okada, H., Wada, T., Yamashita, A., Matsue, T.: Interval-valued evolution strategy for evolving neural networks with interval weights and biases. In: Proceedings of the International Conference on Soft Computing and Intelligent Systems, and the 13th International Symposium on Advanced Intelligence Systems, pp. 2056–2060. IEEE, Kobe (2012)
Desell, T.: Accelerating the evolution of convolutional neural networks with node-level mutations and epigenetic weight initialization. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 157–158. IEEE, Kyoto (2018)
Lyu, Z., ElSaid, A., Karns, J., Mkaouer, M., Desell, T.: An experimental study of weight initialization and weight inheritance effects on neuroevolution. In: Proceedings of Applications of Evolutionary Computation: 24th International Conference. ACM, Seville (2021)
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A.: Evolving deep neural networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293–312 (2019)
Papavasileiou, E., Cornelis, J., Jansen, B.: A systematic literature review of the successors of ‘NeuroEvolution of augmenting topologies’. Evol. Comput. 29, 1–73 (2020)
Bohrer, J.S., Grisci, B.I., Dorn, M.: Neuroevolution of neural network architectures using CoDeepNEAT and Keras. In: arXiv preprint arXiv:2002.04634 (2020)
Zhou, X., Li, X., Hu, K., Zhang, Y., Chen, Z., Gao, X.: ERV-Net: an efficient 3D residual neural network for brain tumor segmentation. Expert Syst. Appl. 170, 114566 (2021)
Dogan, S, et al.: Automated accurate fire detection system using ensemble pretrained residual network. Expert Syst. Appl. 203, 117407 (2022)
Hoorali, F., Khosravi, H., Moradi, B.: IRUNet for medical image segmentation. Expert Syst. Appl. 191, 116399 (2022)
Li, H., Xu, Z., Tyalor, G., Studer, C., Goldstein, T.: Visualizing the loss landscape of neural nets. In: Advances in Neural Information Processing Systems (2018)
Intuitive Explanation of Skip Connections in Deep Learning. https://theaisummer.com/skip-connections/. Accessed 12 June 2022
Keras Documentation - Glorot Uniform. https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotUniform. Accessed 10 July 2022
Keras Documentation - Glorot Normal. https://www.tensorflow.org/api_docs/python/tf/keras/initializers/GlorotNormal. Accessed 10 July 2022
Keras Documentation - He Uniform. https://www.tensorflow.org/api_docs/python/tf/keras/initializers/HeUniform. Accessed 10 July 2022
Keras Documentation - He Normal. https://www.tensorflow.org/api_docs/python/tf/keras/initializers/HeNormal. Accessed 10 July 2022
Searching for activation functions. https://arxiv.org/abs/1710.05941. Accessed 12 June 2022
Mish: A self regularized non-monotonic neural activation function. https://arxiv.org/abs/1908.08681. Accessed 12 June 2022
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Evangelista, L.G.C., Giusti, R. (2022). Short-and-Long-Term Impact of Initialization Functions in NeuroEvolution. In: Xavier-Junior, J.C., Rios, R.A. (eds) Intelligent Systems. BRACIS 2022. Lecture Notes in Computer Science(), vol 13653. Springer, Cham. https://doi.org/10.1007/978-3-031-21686-2_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-21686-2_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21685-5
Online ISBN: 978-3-031-21686-2
eBook Packages: Computer ScienceComputer Science (R0)