Advertisement

Model Selection in Committees of Evolved Convolutional Neural Networks Using Genetic Algorithms

  • Alejandro BaldominosEmail author
  • Yago Saez
  • Pedro Isasi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11314)

Abstract

Neuroevolution is a technique that has been successfully applied for over three decades in order to optimize certain aspects of neural networks by applying evolutionary algorithms. However, only in the recent years, the increase of computational resources has enabled to apply such techniques to deep and convolutional neural networks, where the number of hyperparameters is significantly large.

In recent years, deep and convolutional neural networks are outperforming classical machine learning for many different tasks, including computer vision, natural language processing, signal processing, activity recognition, etc. In this context, neuroevolution can be useful since there are no analytic approaches for determining optimal network architectures or hyperparameters, therefore attaining better performance. Moreover, in some cases, committees (also called ensembles) are used, which combine two or more models in order to improve results even more. Neuroevolution can be of particular importance in this case, since evolutionary algorithms evolve a whole population of individuals, making it easy to build ensembles out of models chosen among those in the evolved population.

In this paper, we explore the application of genetic algorithms for carrying out model selection in a context of neuroevolution. Thus, the best models will be selected from a population of evolved individuals in order to maximize an overall objective function. This approach is tested using the well-known MNIST database as benchmark, and it obtains results which are highly competitive when compared with the state of the art.

Keywords

Genetic algorithms Neuroevolution Convolutional neural network Committees Ensembles 

Notes

Acknowledgement

This research is supported by the Spanish Ministry of Education, Culture and Sport through FPU fellowship with identifier FPU13/03917.

References

  1. 1.
    Akhtyamova, L., Ignatov, A., Cardiff, J.: A Large-scale CNN ensemble for medication safety analysis. In: Frasincar, F., Ittoo, A., Nguyen, L.M., Métais, E. (eds.) NLDB 2017. LNCS, vol. 10260, pp. 247–253. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-59569-6_29CrossRefGoogle Scholar
  2. 2.
    Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning. In: Proceedings of the 5th International Conference on the Learning Repr. (2017)Google Scholar
  3. 3.
    Baldominos, A., Saez, Y., Isasi, P.: Evolutionary convolutional neural networks: an application to handwriting recognition. Neurocomputing 283, 38–52 (2018)CrossRefGoogle Scholar
  4. 4.
    Baldominos, A., Saez, Y., Isasi, P.: Evolutionary design of convolutional neural networks for human activity recognition in sensor-rich environments. Sensors 18(4), 1288 (2018)CrossRefGoogle Scholar
  5. 5.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  6. 6.
    Chang, J.R., Chen, Y.S.: Batch-normalized maxout network in network. arXiv arXiv:1511.02583 (2015)
  7. 7.
    Ciresan, D.C., Meier, U., Gambardella, L.M., Schmidhuber, J.: Convolutional neural network committees for handwritten character classification. In: Proceedings of the 2011 International Conference on Document Analysis and Recognition, pp. 1135–1139 (2011)Google Scholar
  8. 8.
    Davison, J.: DEvol: Automated deep neural network design via genetic programming (2017). https://github.com/joeddav/devol. Accessed 01 July 2017
  9. 9.
    Desell, T.: Large scale evolution of convolutional neural networks using volunteer computing. In: Proceedings 2017 Genetic Evolutionary Computation Conference Companion, pp. 127–128 (2017)Google Scholar
  10. 10.
    Fernando, C., et al.: Convolution by evolution: differentiable pattern producing networks. In: Proceedings 2016 Genetic Evolutionary Computation Conference, pp. 109–116 (2016)Google Scholar
  11. 11.
    Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006)CrossRefGoogle Scholar
  12. 12.
    Hinton, G., Srivastava, N., Swersky, K.: RMSProp: divide the gradient by a running average of its recent magnitude. Neural networks for machine learning, Coursera lecture 6e (2012)Google Scholar
  13. 13.
    Kassahun, Y., Sommer, G.: Efficient reinforcement learning through evolutionary acquisition of neural topologies. In: Proceedings of the 2005 European Symposium on Artificial Neural Networks, pp. 259–266 (2005)Google Scholar
  14. 14.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv arXiv:1412.6980 (2014)
  15. 15.
    Koutník, J., Schmidhuber, J., Gomez, F.: Evolving deep unsupervised convolutional networks for vision-based reinforcement learning. In: Proceedings 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 541–548 (2014)Google Scholar
  16. 16.
    Loshchilov, I., Hutter, F.: CMA-ES for hyperparameter optimization of deep neural networks. arXiv abs/1604.07269 (2016)Google Scholar
  17. 17.
    Miikkulainen, R., et al.: Evolving deep neural networks. arXiv abs/1703.00548 (2017)Google Scholar
  18. 18.
    Ordóñez, F.J., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1), 115 (2016)CrossRefGoogle Scholar
  19. 19.
    Real, E., et al.: Large-scale evolution of image classifiers. arXiv abs/1703.01041 (2017)Google Scholar
  20. 20.
    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)CrossRefGoogle Scholar
  21. 21.
    Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings 2017 Genetic Genetic and Evolutionary Computation Conference, pp. 497–504 (2017)Google Scholar
  22. 22.
    Sultana, J., Rani, M.U., Farquad, M.: An extensive survey on some deep learning applications. In: Proceedings of the 2018 IADS International Conference on Computing, Communications & Data Engineering (2018)Google Scholar
  23. 23.
    Verbancsics, P., Harguess, J.: Image classification using generative neuroevolution for deep learning. In: Proceedings 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 488–493 (2015)Google Scholar
  24. 24.
    Voulodimos, A., Doulamis, N., Doulamis, A., Protopapadakis, E.: Deep learning for computer vision: a brief review. Comput. Intell. Neurosci. 2018, 13 (2018)Google Scholar
  25. 25.
    Xie, L., Yuille, A.: Genetic CNN. arXiv abs/1703.01513 (2017)Google Scholar
  26. 26.
    Yao, X., Liu, Y.: A new evolutionary system for evolving artificial neural networks. IEEE Trans. Neural Netw. 8(3), 694–713 (1997)CrossRefGoogle Scholar
  27. 27.
    Young, S.R., Rose, D.C., Karnowsky, T.P., Lim, S.H., Patton, R.M.: Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: Proceedings Workshop on Machine Learning in High-Performance Computing Environments (2015)Google Scholar
  28. 28.
    Zhang, L., Wang, S., Liu, B.: Deep learning for sentiment analysis: a survey. Interdiscip. Rev. Data Min. Knowl. Discov. 8, e1253 (2018)CrossRefGoogle Scholar
  29. 29.
    Zhang, Q., Yang, L.T., Chen, Z., Li, P.: A survey on deep learning for big data. Inf. Fusion 42, 146–157 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Computer Science DepartmentUniversidad Carlos III de MadridLeganesSpain

Personalised recommendations