Comparison of Neural Network Optimizers for Relative Ranking Retention Between Neural Architectures
- 935 Downloads
Autonomous design and optimization of neural networks is gaining increasingly more attention from the research community. The main barrier is the computational resources required to conduct experimental and production project. Although most researchers focus on new design methodologies, the main computational cost remains the evaluation of candidate architectures. In this paper we investigate the feasibility of using reduced epoch training, by measuring the rank correlation coefficients between sets of optimizers, given a fixed number of training epochs. We discover ranking correlations of more than 0.75 and up to 0.964 between Adam with 50 training epochs, stochastic gradient descent with nesterov momentum with 10 training epochs and Adam with 20 training epochs. Moreover, we show the ability of genetic algorithms to find high-quality solutions of a function, by searching in a perturbed search space, given that certain correlation criteria are met.
KeywordsDeep learning Neural architecture search Ranking
This work was supported by computational time granted from the Greek Research & Technology Network (GRNET) in the National HPC facility - ARIS - under project ID DNAD. Furthermore, this research is funded by the University of Macedonia Research Committee as part of the “Principal Research 2019” funding program.
- 1.Miikkulainen, R., Liang, J., Meyerson, E., et al.: Evolving deep neural networks. In: Artificial Intelligence in the age of Neural Networks and Brain Computing, pp. 293–312 (2019). https://doi.org/10.1016/b978-0-12-815480-9.00015-3CrossRefGoogle Scholar
- 2.Zoph, B., Le, Q.: Neural architecture search with reinforcement learning (2016). https://arxiv.org/abs/1611.01578. Accessed 21 Feb 2019
- 3.Zoph, B., Vasudevan, V., Shlens, J., Le, Q.: Learning transferable architectures for scalable image recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/cvpr.2018.00907
- 4.Kyriakides, G., Margaritis, K.: Neural architecture search with synchronous advantage actor-critic methods and partial training. In: Proceedings of the 10th Hellenic Conference on Artificial Intelligence - SETN 2018 (2018). https://doi.org/10.1145/3200947.3208068
- 7.Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware. https://arxiv.org/abs/1812.00332. Accessed 19 Mar 2019
- 8.Cortes, C., Gonzalvo, X., Kuznetsov, V., et al.: AdaNet: adaptive structural learning of artificial neural networks (2019). https://arxiv.org/abs/1607.01097. Accessed 21 Feb 2019
- 9.Dong, C., Shi, Y., Tao, R.: Convolutional neural networks for clothing image style recognition. DEStech Trans. Comput. Sci. Eng. (2018). https://doi.org/10.12783/dtcse/cmsms2018/25262
- 11.Gatys, L., Ecker, A., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
- 15.Biscani, F., Izzo, D., Jakob, W., et al.: esa/pagmo2: pagmo 2.10. In: Zenodo (2019). https://doi.org/10.5281/zenodo.1045336. Accessed 21 Feb 2019