Global optimization in machine learning: the design of a predictive analytics application
- 126 Downloads
Global optimization, especially Bayesian optimization, has become the tool of choice in hyperparameter tuning and algorithmic configuration to optimize the generalization capability of machine learning algorithms. The contribution of this paper was to extend this approach to a complex algorithmic pipeline for predictive analytics, based on time-series clustering and artificial neural networks. The software environment R has been used with mlrMBO, a comprehensive and flexible toolbox for sequential model-based optimization. Random forest has been adopted as surrogate model, due to the nature of decision variables (i.e., conditional and discrete hyperparameters) of the case studies considered. Two acquisition functions have been considered: Expected improvement and lower confidence bound, and results are compared. The computational results, on a benchmark and a real-world dataset, show that even in a complex search space, up to 80 dimensions related to integer, categorical, and conditional variables (i.e., hyperparameters), sequential model-based optimization is an effective solution, with lower confidence bound requiring a lower number of function evaluations than expected improvement to find the same optimal solution.
KeywordsHyperparameters optimization Global optimization Machine learning
Compliance with ethical standards
Conflict of interest
Antonio Candelieri and Francesco Archetti declare that they have no conflict of interest.
This article does not contain any studies with human participants or animals performed by any of the authors.
- Bischl B, Richter J, Bossek J, Horn D, Thomas J, Lang M (2017) mlrMBO: a modular framework for model-based optimization of expensive black-box functions. arXiv:1703.03373
- Candelieri A, Giordani I, Archetti F (2017) Automatic configuration of kernel-based clustering: an optimization approach. In: International conference on learning and intelligence optimization. Springer, Cham, pp 34–49Google Scholar
- Dhillon IS, Guan Y, Kulis B (2004) Kernel k-means: spectral clustering and normalized cuts. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining, pp 551–556Google Scholar
- Feurer M, Klein A, Eggensperger K, Springenberg J, Blum M, Hutter F (2015) Efficient and robust automated machine learning. In: Advances in neural information processing systems, pp 2962–2970Google Scholar
- Kandasamy K, Schneider J, Pòczos B (2015) High dimensional Bayesian optimisation and bandits via additive models. In: International conference on machine learning, vol 37, pp 295–304Google Scholar
- Mockus J, Tiesis V, Zilinskas A (1978) The application of Bayesian methods for seeking the extremum. In: Dixon L, Szego G (eds) Towards global optimisation 2. Elsevier, New York, pp 117–130Google Scholar
- Snoek J, Larochelle H, Adams RP (2012) Practical Bayesian optimization of machine learning algorithms. arXiv:1206.2944[stat.ML]
- Thornton C, Hutter F, Hoos HH, Leyton-Brown K (2013) Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In: Proceedings of ACM SIGKDD, pp 847–855Google Scholar
- Wang Z, Zoghi M, Hutter F, Matheson D, De Freitas N (2013) Bayesian optimization in high dimensions via random embeddings. In: Proceedings of the international joint conference on artificial intelligence, pp 1778–1784Google Scholar