Model Selection in Committees of Evolved Convolutional Neural Networks Using Genetic Algorithms
Neuroevolution is a technique that has been successfully applied for over three decades in order to optimize certain aspects of neural networks by applying evolutionary algorithms. However, only in the recent years, the increase of computational resources has enabled to apply such techniques to deep and convolutional neural networks, where the number of hyperparameters is significantly large.
In recent years, deep and convolutional neural networks are outperforming classical machine learning for many different tasks, including computer vision, natural language processing, signal processing, activity recognition, etc. In this context, neuroevolution can be useful since there are no analytic approaches for determining optimal network architectures or hyperparameters, therefore attaining better performance. Moreover, in some cases, committees (also called ensembles) are used, which combine two or more models in order to improve results even more. Neuroevolution can be of particular importance in this case, since evolutionary algorithms evolve a whole population of individuals, making it easy to build ensembles out of models chosen among those in the evolved population.
In this paper, we explore the application of genetic algorithms for carrying out model selection in a context of neuroevolution. Thus, the best models will be selected from a population of evolved individuals in order to maximize an overall objective function. This approach is tested using the well-known MNIST database as benchmark, and it obtains results which are highly competitive when compared with the state of the art.
KeywordsGenetic algorithms Neuroevolution Convolutional neural network Committees Ensembles
This research is supported by the Spanish Ministry of Education, Culture and Sport through FPU fellowship with identifier FPU13/03917.
- 2.Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning. In: Proceedings of the 5th International Conference on the Learning Repr. (2017)Google Scholar
- 6.Chang, J.R., Chen, Y.S.: Batch-normalized maxout network in network. arXiv arXiv:1511.02583 (2015)
- 7.Ciresan, D.C., Meier, U., Gambardella, L.M., Schmidhuber, J.: Convolutional neural network committees for handwritten character classification. In: Proceedings of the 2011 International Conference on Document Analysis and Recognition, pp. 1135–1139 (2011)Google Scholar
- 8.Davison, J.: DEvol: Automated deep neural network design via genetic programming (2017). https://github.com/joeddav/devol. Accessed 01 July 2017
- 9.Desell, T.: Large scale evolution of convolutional neural networks using volunteer computing. In: Proceedings 2017 Genetic Evolutionary Computation Conference Companion, pp. 127–128 (2017)Google Scholar
- 10.Fernando, C., et al.: Convolution by evolution: differentiable pattern producing networks. In: Proceedings 2016 Genetic Evolutionary Computation Conference, pp. 109–116 (2016)Google Scholar
- 12.Hinton, G., Srivastava, N., Swersky, K.: RMSProp: divide the gradient by a running average of its recent magnitude. Neural networks for machine learning, Coursera lecture 6e (2012)Google Scholar
- 13.Kassahun, Y., Sommer, G.: Efficient reinforcement learning through evolutionary acquisition of neural topologies. In: Proceedings of the 2005 European Symposium on Artificial Neural Networks, pp. 259–266 (2005)Google Scholar
- 14.Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv arXiv:1412.6980 (2014)
- 15.Koutník, J., Schmidhuber, J., Gomez, F.: Evolving deep unsupervised convolutional networks for vision-based reinforcement learning. In: Proceedings 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 541–548 (2014)Google Scholar
- 16.Loshchilov, I., Hutter, F.: CMA-ES for hyperparameter optimization of deep neural networks. arXiv abs/1604.07269 (2016)Google Scholar
- 17.Miikkulainen, R., et al.: Evolving deep neural networks. arXiv abs/1703.00548 (2017)Google Scholar
- 19.Real, E., et al.: Large-scale evolution of image classifiers. arXiv abs/1703.01041 (2017)Google Scholar
- 21.Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: Proceedings 2017 Genetic Genetic and Evolutionary Computation Conference, pp. 497–504 (2017)Google Scholar
- 22.Sultana, J., Rani, M.U., Farquad, M.: An extensive survey on some deep learning applications. In: Proceedings of the 2018 IADS International Conference on Computing, Communications & Data Engineering (2018)Google Scholar
- 23.Verbancsics, P., Harguess, J.: Image classification using generative neuroevolution for deep learning. In: Proceedings 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 488–493 (2015)Google Scholar
- 24.Voulodimos, A., Doulamis, N., Doulamis, A., Protopapadakis, E.: Deep learning for computer vision: a brief review. Comput. Intell. Neurosci. 2018, 13 (2018)Google Scholar
- 25.Xie, L., Yuille, A.: Genetic CNN. arXiv abs/1703.01513 (2017)Google Scholar
- 27.Young, S.R., Rose, D.C., Karnowsky, T.P., Lim, S.H., Patton, R.M.: Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: Proceedings Workshop on Machine Learning in High-Performance Computing Environments (2015)Google Scholar