Abstract
Deep learning is a field in artificial intelligence that works well in computer vision, natural language processing and audio recognition. Deep neural network architectures has number of layers to conceive the features well, by itself. The hyperparameter tuning plays a major role in every dataset which has major effect in the performance of the training model. Due to the large dimensionality of data it is impossible to tune the parameters by human expertise. In this paper, we have used the CIFAR-10 Dataset and applied the Bayesian hyperparameter optimization algorithm to enhance the performance of the model. Bayesian optimization can be used for any noisy black box function for hyperparameter tuning. In this work Bayesian optimization clearly obtains optimized values for all hyperparameters which saves time and improves performance. The results also show that the error has been reduced in graphical processing unit than in CPU by 6.2% in the validation. Achieving global optimization in the trained model helps transfer learning across domains as well.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Angelov P (1994) A generalized approach to fuzzy optimization. Int J Intell Syst 9(3):261–268. https://doi.org/10.1002/int.4550090302
Angelov P, Sadeghi-Tehran P, Ramezani R (2011) An approach to automatic real-time novelty detection, object identification, and tracking in video streams based on recursive density estimation and evolving Takagi-Sugeno fuzzy systems. Spec Issue Intell Data Process Methodol Prob Realiz Trends 26(3):189–205. https://doi.org/10.1002/int.20462
Baruah RD, Angelov P (2014) DEC: dynamically evolving clustering and its application to structure identification of evolving fuzzy models. IEEE Trans Cybernet 44(9):1619–1631. https://doi.org/10.1109/TCYB.2013.2291234
Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13:281–305. http://www.jmlr.org/papers/v13/bergstra12a.html
Bergstra J, Bardenet R, Bengio Y, Kegl B (2011) Algorithms for hyper-parameter optimization. Adv Neural Inf Process Syst. https://doi.org/10.5555/2986459.2986743
Bergstra J, Yamins D, Cox DD (2013) Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Proceedings of the 30th International Conference on Machine learning, Atlanta, Georgia, USA. https://doi.org/10.5555/3042817.3042832
Brochu E, Cora VM, de Freitas N (2009) A tutorial on Bayesian optimization of expensive cost functions, with application to active user modelling and hierarchical reinforcement learning. Technical Report UBC TR-2009-23, Dept. of Computer Science, University of British Columbia. https://arXiv.org/abs/1012.2599v1
Calandra R, Seyfarth A, Peters J, Deisenroth MP (2016) Bayesian optimization for learning gaits under uncertainty an experimental comparison on a dynamic bipedal walker. Ann Math Artif Intell 76(1–2):5–23. https://doi.org/10.1007/s10472-015-9463-9
Dewancker I, McCourt M, Clark S (2015) Bayesian optimization primer. https://app.sigopt.com/static/pdf/SigOpt_Bayesian_Optimization_Primer.pdf
Erhan D, Szegedy C, Toshev A, Anguelov D (2014) Scalable object detection using deep neural networks. In Proc. IEEE Conf. on computer vision and pattern recognition, pp 2147–2154. https://arXiv.org/abs/1312.2249v1
Feurer M, Klein A, Eggensperger K, Springenberg JT, Blum M, Hutter F (2019) Auto-sklearn: efficient and robust automated machine learning, part of the springer series on challenges in machine learning book series (SSCML). https://doi.org/10.1007/978-3-030-05318-5_6
Guiming D, Xia W, Guangyan W, Yan Z, Dan L (2016) Speech recognition based on convolutional neural networks. In: IEEE international conference on signal and image processing (ICSIP), Beijing, pp 708–711. https://doi.org/10.1109/SIPROCESS.2016.7888355
Hasanpour SH, Rouhani M, Fayyaz M, Sabokrou M (2018) Let’s keep it simple: using simple architectures to outperform deeper and more complex architectures. https://arxiv.org/abs/1608.06037v7
Hoffman M, Shahriari B, de Freitas N (2014) On correlation and budget constraints in model-based bandit optimization with application to automatic machine learning. AI Stat. https://proceedings.mlr.press/v33/hoffman14.pdf
Hutter F, Hoos HH, Leyton-Brown K (2014) Sequential model-based optimization for general algorithm configuration. LION. https://doi.org/10.1007/978-3-642-25566-3_40
Joy TT, Rana S, Gupta S, Venkatesh S (2016) Hyperparameter tuning for big data using Bayesian optimisation. Proc IEEE Int Conf Pattern Recogn. https://doi.org/10.1109/ICPR.2016.7900023
Knudde N et al (2018) Data-efficient Bayesian optimization with constraints for power amplifier design. In: 2018 IEEE MTT-S international conference on numerical electromagnetic and multiphysics modelling and optimization (NEMO), Reykjavik, pp. 1–3. https://doi.org/10.1109/nemo.2018.8503107
Kochanski G, Golovin D, Karro J, Solnik B, Moitra S, Sculley D (2017) Bayesian optimization for a better dessert. In: 31st conference on neural information processing systems (NIPS) Long Beach, CA, USA. https://www.researchgate.net/publication/324273062_Bayesian_Optimization_for_a_Better_Dessert
Kramer O, Ciaurri DE, Koziel S (2011) Derivative-free optimization. In: Computational optimization, methods and algorithms. Springer, pp. 61–83. https://doi.org/10.1007/978-3-642-20859-1_4
Le Cun Y, Yoshua B, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539
Le HT, Phung SL, Bouzerdoum A, Tivive FHC (2018) Human motion classification with micro-doppler radar and Bayesian-optimized convolutional neural networks. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), Calgary, AB, 2018, pp 2961–2965. https://doi.org/10.1109/icassp.2018.8461847
Lorenzo PR et al (2017) Hyper-parameter selection in deep neural networks using parallel particle swarm optimization. In: Proceedings of the genetic and evolutionary computation conference companion. ACM https://doi.org/10.1145/3067695.3084211
Lyu W et al (2018) An efficient Bayesian optimization approach for automated optimization of analog circuits. IEEE Trans Circuits Syst I Regul Papers 65(6):1954–1967. https://doi.org/10.1109/TCSI.2017.2768826
Mantovani RG, Rossit ALD, Vanschoent J, Bischl B, Carvalho ACPLF (2015) To tune or not to tune: recommending when to adjust SVM hyper-parameters via meta-learning. In: IEEE Proceedings of the international joint conference on neural networks, At Killarney, Ireland. https://doi.org/10.1109/ijcnn.2015.7280644
Marchant R, Ramos F (2012) Bayesian optimisation for intelligent environmental monitoring. Proceedings of the IEEE/RSJ international conference on intelligent robots and systems. IEEE/RSJ Int Conf Intell Robots Syst. https://doi.org/10.1109/iros.2012.6385653
Shahriari B, Swersky K, Wang Z, Adams RP, de Freitas N (2016) Taking the human out of the loop: a review of Bayesian optimization. Proc IEEE. https://doi.org/10.1109/JPROC.2015.2494218
Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. Adv Neural Inf Process Syst. https://arxiv.org/pdf/1206.2944.pdf
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Victoria, A.H., Maragatham, G. Automatic tuning of hyperparameters using Bayesian optimization. Evolving Systems 12, 217–223 (2021). https://doi.org/10.1007/s12530-020-09345-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12530-020-09345-2