Fast DENSER: Efficient Deep NeuroEvolution

  • Filipe AssunçãoEmail author
  • Nuno Lourenço
  • Penousal Machado
  • Bernardete Ribeiro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11451)


The search for Artificial Neural Networks (ANNs) that are effective in solving a particular task is a long and time consuming trial-and-error process where we have to make decisions about the topology of the network, learning algorithm, and numerical parameters. To ease this process, we can resort to methods that seek to automatically optimise either the topology or simultaneously the topology and learning parameters of ANNs. The main issue of such approaches is that they require large amounts of computational resources, and take a long time to generate a solution that is considered acceptable for the problem at hand. The current paper extends Deep Evolutionary Network Structured Representation (DENSER): a general-purpose NeuroEvolution (NE) approach that combines the principles of Genetic Algorithms with Grammatical Evolution; to adapt DENSER to optimise networks of different structures, or to solve various problems the user only needs to change the grammar that is specified in a text human-readable format. The new method, Fast DENSER (F-DENSER), speeds up DENSER, and adds another representation-level that allows the connectivity of the layers to be evolved. The results demonstrate that F-DENSER has a speedup of 20 times when compared to the time DENSER takes to find the best solutions. Concerning the effectiveness of the approach, the results are highly competitive with the state-of-the-art, with the best performing network reporting an average test accuracy of 91.46% on CIFAR-10. This is particularly remarkable since the reduction in the running time does not compromise the performance of the generated solutions.


Automatic Machine Learning Convolutional Neural Networks NeuroEvolution 



This work is partially funded by: Fundação para a Ciência e Tecnologia (FCT), Portugal, under the PhD grant SFRH/BD/114865/2016, and the project grant DSAIPA/DS/0022/2018 (GADgET), and is based upon work from COST Action CA15140: ImAppNIO, supported by COST (European Cooperation in Science and Technology):


  1. 1.
    Assunção, F., Lourenço, N., Machado, P., Ribeiro, B.: Evolving the topology of large scale deep neural networks. In: Castelli, M., Sekanina, L., Zhang, M., Cagnoni, S., García-Sánchez, P. (eds.) Genetic Programming, pp. 19–34. Springer International Publishing, Cham (2018). Scholar
  2. 2.
    Assunção, F., Lourenço, N., Machado, P., Ribeiro, B.: DENSER: deep evolutionary network structured representation. Genet. Program. Evolvable Mach. (2018).
  3. 3.
    Guyon, I., et al.: A brief review of the ChaLearn AutoML challenge: any-time any-dataset learning without human intervention. In: AutoML@ICML. JMLR Workshop and Conference Proceedings, vol. 64, pp. 21–30 (2016)Google Scholar
  4. 4.
    Duan, K.-B., Keerthi, S.S.: Which is the best multiclass SVM method? An empirical study. In: Oza, N.C., Polikar, R., Kittler, J., Roli, F. (eds.) MCS 2005. LNCS, vol. 3541, pp. 278–285. Springer, Heidelberg (2005). Scholar
  5. 5.
    Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Bergstra, J., Yamins, D., Cox, D.D.: Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In: JMLR Workshop and Conference Proceedings, ICML (1), vol. 28, pp. 115–123 (2013)Google Scholar
  7. 7.
    Miikkulainen, R., et al.: Evolving deep neural networks. CoRR abs/1703.00548 (2017)Google Scholar
  8. 8.
    Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: from architectures to learning. Evol. Intell. 1(1), 47–62 (2008)CrossRefGoogle Scholar
  9. 9.
    Koutník, J., Cuccu, G., Schmidhuber, J., Gomez, F.J.: Evolving large-scale neural networks for vision-based TORCS. In: FDG, pp. 206–212. Society for the Advancement of the Science of Digital Games (2013)Google Scholar
  10. 10.
    Gomez, F.J., Schmidhuber, J., Miikkulainen, R.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 9, 937–965 (2008)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Turner, A.J., Miller, J.F.: The importance of topology evolution in neuroevolution: a case study using cartesian genetic programming of artificial neural networks. In: Bramer, M., Petridis, M. (eds.) Research and Development in Intelligent Systems XXX, pp. 213–226. Springer, Cham (2013). Scholar
  12. 12.
    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRefGoogle Scholar
  13. 13.
    Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 497–504. ACM (2017)Google Scholar
  14. 14.
    Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548 (2018)
  15. 15.
    Lourenço, N., Assunção, F., Pereira, F.B., Costa, E., Machado, P.: Structured grammatical evolution: a dynamic approach. In: Ryan, C., O’Neill, M., Collins, J. (eds.) Handbook of Grammatical Evolution, pp. 137–161. Springer, Cham (2018). Scholar
  16. 16.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.CISUC, Department of Informatics EngineeringUniversity of CoimbraCoimbraPortugal

Personalised recommendations