Evolving the Topology of Large Scale Deep Neural Networks

  • Filipe AssunçãoEmail author
  • Nuno Lourenço
  • Penousal Machado
  • Bernardete Ribeiro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10781)


In the recent years Deep Learning has attracted a lot of attention due to its success in difficult tasks such as image recognition and computer vision. Most of the success in these tasks is merit of Convolutional Neural Networks (CNNs), which allow the automatic construction of features. However, designing such networks is not an easy task, which requires expertise and insight. In this paper we introduce DENSER, a novel representation for the evolution of deep neural networks. In concrete we adapt ideas from Genetic Algorithms (GAs) and Grammatical Evolution (GE) to enable the evolution of sequences of layers and their parameters. We test our approach in the well-known image classification CIFAR-10 dataset. The results show that our method: (i) outperforms previous evolutionary approaches to the generations of CNNs; (ii) is able to create CNNs that have state-of-the-art performance while using less prior knowledge (iii) evolves CNNs with novel topologies, unlikely to be designed by hand. For instance, the best performing CNN obtained during evolution has an unexpected structure using six consecutive dense layers. On the CIFAR-10 the best model reports an average error of 5.87% on test data.


Convolutional Neural Networks Deep Neural Networks Genetic Algorithm Dynamic Structured Grammatical Evolution 



This work is partially funded by: Fundação para a Ciência e Tecnologia (FCT), Portugal, under the grant SFRH/BD/114865/2016, and is based upon work from COST Action CA15140: ImAppNIO, supported by COST (European Cooperation in Science and Technology): We would also like to thank NVIDIA for providing us Titan X GPUs.


  1. 1.
    Assunção, F., Lourenço, N., Machado, P., Ribeiro, B.: Towards the evolution of multi-layered neural networks: a dynamic structured grammatical evolution approach. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 393–400. ACM, New York (2017).
  2. 2.
    Blum, A., Rivest, R.L.: Training a 3-node neural network is NP-complete. In: Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 494–501. MIT Press (1988)Google Scholar
  3. 3.
    Buk, Z., Koutník, J., Šnorek, M.: NEAT in HyperNEAT substituted with genetic programming. In: Kolehmainen, M., Toivanen, P., Beliczynski, B. (eds.) ICANNGA 2009. LNCS, vol. 5495, pp. 243–252. Springer, Heidelberg (2009). CrossRefGoogle Scholar
  4. 4.
    David, O.E., Greental, I.: Genetic algorithms for evolving deep neural networks. In: Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 1451–1452. ACM (2014)Google Scholar
  5. 5.
    Deb, K., Anand, A., Joshi, D.: A computationally efficient evolutionary algorithm for real-parameter optimization. Evol. Comput. 10(4), 371–395 (2002)CrossRefGoogle Scholar
  6. 6.
    Deng, L., Hinton, G., Kingsbury, B.: New types of deep neural network learning for speech recognition and related applications: an overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8599–8603. IEEE (2013)Google Scholar
  7. 7.
    Farfade, S.S., Saberian, M.J., Li, L.J.: Multi-view face detection using deep convolutional neural networks. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, ICMR 2015, pp. 643–650. ACM, New York (2015).
  8. 8.
    Franco, L., Jerez, J.M.: Constructive Neural Networks, vol. 258. Springer, Heidelberg (2009). CrossRefGoogle Scholar
  9. 9.
    Gomez, F., Schmidhuber, J., Miikkulainen, R.: Accelerated neural evolution through cooperatively coevolved synapses. J. Mach. Learn. Res. 9(May), 937–965 (2008)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Graham, B.: Fractional max-pooling. arXiv preprint arXiv:1412.6071 (2014)
  11. 11.
    Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649, May 2013Google Scholar
  12. 12.
    Junyou, B.: Stock price forecasting using PSO-trained neural networks. In: 2007 IEEE Congress on Evolutionary Computation, CEC 2007, pp. 2879–2885. IEEE (2007)Google Scholar
  13. 13.
    Kim, H.B., Jung, S.H., Kim, T.G., Park, K.H.: Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates. Neurocomputing 11(1), 101–106 (1996)CrossRefzbMATHGoogle Scholar
  14. 14.
    Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)Google Scholar
  15. 15.
    Leung, F.H.F., Lam, H.K., Ling, S.H., Tam, P.K.S.: Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Trans. Neural Netw. 14(1), 79–88 (2003)CrossRefGoogle Scholar
  16. 16.
    Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. arXiv preprint arXiv:1703.00548 (2017)
  17. 17.
    Mishkin, D., Matas, J.: All you need is a good init. arXiv preprint arXiv:1511.06422 (2015)
  18. 18.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)CrossRefGoogle Scholar
  19. 19.
    Moriarty, D.E., Miikkulainen, R.: Forming neural networks through efficient and adaptive coevolution. Evol. Comput. 5(4), 373–399 (1997)CrossRefGoogle Scholar
  20. 20.
    O’Neil, M., Ryan, C.: Grammatical evolution. In: O’Neil, M., Ryan, C. (eds.) Grammatical Evolution, pp. 33–47. Springer, Boston (2003). CrossRefGoogle Scholar
  21. 21.
    Palmes, P.P., Hayasaka, T., Usui, S.: Evolution and adaptation of neural networks. In: 2003 Proceedings of the International Joint Conference on Neural Networks, vol. 1, pp. 478–483. IEEE (2003)Google Scholar
  22. 22.
    Parra, J., Trujillo, L., Melin, P.: Hybrid back-propagation training with evolutionary strategies. Soft. Comput. 18(8), 1603–1614 (2014)CrossRefGoogle Scholar
  23. 23.
    Plis, S.M., Hjelm, D.R., Salakhutdinov, R., Allen, E.A., Bockholt, H.J., Long, J.D., Johnson, H.J., Paulsen, J.S., Turner, J.A., Calhoun, V.D.: Deep learning for neuroimaging: a validation study. Front. Neurosci. 8, 229 (2014)CrossRefGoogle Scholar
  24. 24.
    Radi, A., Poli, R.: Discovering efficient learning rules for feedforward neural networks using genetic programming. In: Abraham, A., Jain, L.C., Kacprzyk, J. (eds.) Recent Advances in Intelligent Paradigms and Applications, pp. 133–159. Springer, Heidelberg (2003). CrossRefGoogle Scholar
  25. 25.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  26. 26.
    Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., Adams, R.: Scalable Bayesian optimization using deep neural networks. In: International Conference on Machine Learning, pp. 2171–2180 (2015)Google Scholar
  27. 27.
    Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
  28. 28.
    Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)CrossRefGoogle Scholar
  29. 29.
    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRefGoogle Scholar
  30. 30.
    Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, pp. 497–504. ACM, New York (2017).
  31. 31.
    Whitley, D., Starkweather, T., Bogart, C.: Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput. 14(3), 347–361 (1990)CrossRefGoogle Scholar
  32. 32.
    Yao, X., Liu, Y.: Evolutionary artificial neural networks that learn and generalise well. In: 1996 IEEE International Conference on Neural Networks, Washington, DC, USA, Volume on Plenary, Panel and Special Sessions, pp. 159–164 (1996)Google Scholar
  33. 33.
    Zhang, J., Zong, C.: Deep neural networks in machine translation: an overview. IEEE Intell. Syst. 30(5), 16–25 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.CISUC, Department of Informatics EngineeringUniversity of CoimbraCoimbraPortugal

Personalised recommendations