Advertisement

Machine Learning

, Volume 107, Issue 1, pp 15–41 | Cite as

Efficient benchmarking of algorithm configurators via model-based surrogates

  • Katharina Eggensperger
  • Marius Lindauer
  • Holger H. Hoos
  • Frank Hutter
  • Kevin Leyton-Brown
Article
  • 1k Downloads
Part of the following topical collections:
  1. Special Issue on Metalearning and Algorithm Selection

Abstract

The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. However, the proper evaluation of new algorithm configuration (AC) procedures (or configurators) is hindered by two key hurdles. First, AC scenarios are hard to set up, including the target algorithm to be optimized and the problem instances to be solved. Second, and even more significantly, they are computationally expensive: a single configurator run involves many costly runs of the target algorithm. Here, we propose a benchmarking approach that uses surrogate scenarios, which are computationally cheap while remaining close to the original AC scenarios. These surrogate scenarios approximate the response surface corresponding to true target algorithm performance using a regression model. In our experiments, we construct and evaluate surrogate scenarios for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems. We generalize previous work by building surrogates for AC scenarios with multiple problem instances, stochastic target algorithms and censored running time observations. We show that our surrogate scenarios capture overall important characteristics of the original AC scenarios from which they were derived, while being much easier to use and orders of magnitude cheaper to evaluate.

Keywords

Algorithm configuration Hyper-parameter optimization Empirical performance model 

Notes

Acknowledgements

We thank Stefan Falkner for the implementation of the quantile regression forest used in our experiments and for fruitful discussions on early drafts of the paper. K. Eggensperger, M. Lindauer and F. Hutter acknowledge funding by the DFG (German Research Foundation) under Emmy Noether Grant HU 1900/2-1; K. Eggensperger also acknowledges funding by the State Graduate Funding Program of Baden-Württemberg. H. Hoos and K. Leyton-Brown acknowledge funding through NSERC Discovery Grants; K. Leyton-Brown also acknowledges funding from an NSERC E.W.R. Steacie Fellowship.

References

  1. Ahmadizadeh, K., Dilkina, B., Gomes, C., & Sabharwal, A. (2010). An empirical study of optimization for maximizing diffusion in networks. In D. Cohen (Ed.) Proceedings of the international conference on principles and practice of constraint programming (CP’10), Lecture Notes in Computer Science (Vol. 6308, pp. 514–521). Berlin: Springer.Google Scholar
  2. Ansel, J., Kamil, S., Veeramachaneni, K., Ragan-Kelley, J., Bosboom, J., O’Reilly, U., & Amarasinghe, S. (2014). Opentuner: An extensible framework for program autotuning. In J. Amaral, & J. Torrellas (Eds.) International conference on parallel architectures and compilation (pp. 303–316). New York: ACM.Google Scholar
  3. Ansótegui, C., Sellmann, M., & Tierney, K. (2009). A gender-based genetic algorithm for the automatic configuration of algorithms. In I. Gent (Ed.) Proceedings of the fifteenth international conference on principles and practice of constraint programming (CP’09) Lecture Notes in Computer Science (Vol. 5732, pp. 142–157). Berlin: Springer.Google Scholar
  4. Ansótegui, C., Malitsky, Y., Sellmann, M., & Tierney, K. (2015). Model-based genetic algorithms for algorithm configuration. In Q. Yang, & M. Wooldridge (Eds.) Proceedings of the 25th international joint conference on artificial intelligence (IJCAI’15) (pp. 733–739).Google Scholar
  5. Arbelaez, A., Truchet, C., & O’Sullivan, B. (2016). Learning sequential and parallel runtime distributions for randomized algorithms. In Proceedings of the international conference on tools with artificial intelligence (ICTAI).Google Scholar
  6. Balint, A., & Schöning, U. (2012). Choosing probability distributions for stochastic local search and the role of make versus break. In A. Cimatti, & R. Sebastiani (Eds.) Proceedings of the fifteenth international conference on theory and applications of satisfiability testing (SAT’12), Lecture Notes in Computer Science (Vol. 7317, pp. 16–29). Berlin:Springer.Google Scholar
  7. Bardenet, R., Brendel, M., Kégl, B., & Sebag, M. (2014). Collaborative hyperparameter tuning. In S. Dasgupta, & D. McAllester (Eds.) Proceedings of the 30th international conference on machine learning (ICML’13) (pp. 199–207). Madison: Omnipress.Google Scholar
  8. Bensusan, H., & Kalousis, A. (2001). Estimating the predictive accuracy of a classifier. In Proceedings of the 12th European conference on machine learning (ECML) (pp. 25–36). Berlin: Springer.Google Scholar
  9. Bergstra, J., Yamins, D., & Cox, D. (2014). Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In S. Dasgupta, & D. McAllester (Eds.) Proceedings of the 30th international conference on machine learning (ICML’13) (pp. 115–123). Madison: Omnipress.Google Scholar
  10. Biere, A. (2013). Lingeling, plingeling and treengeling entering the sat competition 2013. In A. Balint, A. Belov, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2013: Solver and benchmark descriptions (Vol. B-2013-1, pp. 51–52). Helsinki: University of Helsinki, Department of Computer Science Series of Publications B.Google Scholar
  11. Biere, A. (2014). Yet another local search solver and Lingeling and friends entering the SAT competition 2014. In A. Belov, D. Diepold, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2014: Solver and benchmark descriptions (Vol. B-2014-2, pp. 39–40). Helsinki: University of Helsinki, Department of Computer Science Series of Publications B.Google Scholar
  12. Birattari, M., Stützle, T., Paquete, L., & Varrentrapp, K. (2002). A racing algorithm for configuring metaheuristics. In W. Langdon, E. Cantu-Paz, K. Mathias, R. Roy, D. Davis, R. Poli, K. Balakrishnan, V. Honavar, G. Rudolph, J. Wegener, L. Bull, M. Potter, A. Schultz, J. Miller, E. Burke, & N. Jonoska (Eds.) Proceedings of the genetic and evolutionary computation conference (GECCO’02) (pp. 11–18). Burlington: Morgan Kaufmann Publishers.Google Scholar
  13. Bischl, B., Kerschke, P., Kotthoff, L., Lindauer, M., Malitsky, Y., Frechétte, A., et al. (2016). ASlib: A benchmark library for algorithm selection. Artificial Intelligence, 237, 41–58.MathSciNetCrossRefMATHGoogle Scholar
  14. Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight uncertainty in neural network. In F. Bach, & D. Blei (Eds.) Proceedings of the 32nd international conference on machine learning (ICML’15) (Vol. 37, pp. 1613–1622). Madison: Omnipress.Google Scholar
  15. Brazdil, P., Giraud-Carrier, C., Soares, C., & Vilalta, R. (2008). Metalearning: Applications to data mining (1st ed.). Berlin: Springer Publishing Company.MATHGoogle Scholar
  16. Breimann, L. (2001). Random forests. Machine Learning Journal, 45, 5–32.CrossRefGoogle Scholar
  17. Brochu, E., Cora, V., & de Freitas, N. (2010). A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Computing Research Repository (CoRR) abs/1012.2599.Google Scholar
  18. Brummayer, R., Lonsing, F., & Biere, A. (2012). Automated testing and debugging of SAT and QBF solvers. In A. Cimatti, & R. Sebastiani (Eds.) Proceedings of the fifteenth international conference on theory and applications of satisfiability testing (SAT’12), Lecture Notes in Computer Science (Vol. 7317, pp. 44–57). Berlin: Springer.Google Scholar
  19. Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In B. Krishnapuram, M. Shah, A. Smola, C. Aggarwal, D. Shen, & R. Rastogi (Eds.) Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (KDD) (pp. 785–794). New York: ACM.Google Scholar
  20. Collobert, R., Bengio, S., & Bengio, Y. (2002). A parallel mixture of svms for very large scale problems. Neural Computation, 14(5), 1105–1114.CrossRefMATHGoogle Scholar
  21. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.MATHGoogle Scholar
  22. Dixon, L., & Szegö, G. (1978). The global optimization problem: An introduction. Towards global optimization, 2, 1–15.Google Scholar
  23. Domhan, T., Springenberg, J. T., & Hutter, F. (2015). Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Q. Yang, & M. Wooldridge (Eds.) Proceedings of the 25th international joint conference on artificial intelligence (IJCAI’15) (pp 3460–3468).Google Scholar
  24. Eén, N., & Sörensson, N. (2003). An extensible sat-solver. In E. Giunchiglia, & A. Tacchella (Eds.) Proceedings of the conference on theory and applications of satisfiability testing (SAT) Lecture Notes in Computer Science (Vol. 2919, pp. 502–518). Berlin: Springer.Google Scholar
  25. Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H., & Leyton-Brown, K. (2013). Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS workshop on Bayesian optimization in theory and practice (BayesOpt’13).Google Scholar
  26. Eggensperger, K., Hutter, F., Hoos, H., & Leyton-Brown, K. (2015). Efficient benchmarking of hyperparameter optimizers via surrogates. In B. Bonet, & S. Koenig (Eds.) Proceedings of the twenty-nineth national conference on artificial intelligence (AAAI’15) (pp. 1114–1120). AAAI Press.Google Scholar
  27. Fawcett, C., & Hoos, H. (2016). Analysing differences between algorithm configurations through ablation. Journal of Heuristics, 22(4), 431–458.CrossRefGoogle Scholar
  28. Fawcett, C., Vallati, M., Hutter, F., Hoffmann, J., Hoos, H., & Leyton-Brown, K. (2014). Improved features for runtime prediction of domain-independent planners. In S. Chien, D. Minh, A. Fern, & W. Ruml (Eds.) Proceedings of the twenty-fourth international conference on automated planning and scheduling (ICAPS-14), AAAI.Google Scholar
  29. Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., & Blum, M., Hutter, F. (2015a). Efficient and robust automated machine learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, & R. Garnett (Eds.) Proceedings of the 29th international conference on advances in neural information processing systems (NIPS’15) (pp. 2962–2970).Google Scholar
  30. Feurer, M., Springenberg, T., & Hutter, F. (2015b). Initializing Bayesian hyperparameter optimization via meta-learning. In B. Bonet, & S. Koenig (Eds.) Proceedings of the twenty-nineth national conference on artificial intelligence (AAAI’15) (pp. 1128–1135). AAAI Press.Google Scholar
  31. Gama, J., & Brazdil, P. (1995). Characterization of classification algorithms. In Proceedings of the 7th Portuguese conference on artificial intelligence (pp. 189–200). Berlin: Springer.Google Scholar
  32. Gebser, M., Kaminski, R., Kaufmann, B., Schaub, T., Schneider, M., & Ziller, S. (2011). A portfolio solver for answer set programming: Preliminary report. In J. Delgrande, & W. Faber (Eds.) Proceedings of the eleventh international conference on logic programming and nonmonotonic reasoning (LPNMR’11), Lecture Notes in Computer Science (Vol. 6645, pp. 352–357). Berlin: Springer.Google Scholar
  33. Gebser, M., Kaufmann, B., & Schaub, T. (2012). Conflict-driven answer set solving: From theory to practice. Artificial Intelligence, 187–188, 52–89.MathSciNetCrossRefMATHGoogle Scholar
  34. Gelbart, M., Snoek, J., & Adams, R. (2014). Bayesian optimization with unknown constraints. In N. Zhang, & J. Tian (Eds.) Proceedings of the 30th conference on uncertainty in artificial intelligence (UAI’14), AUAI Press.Google Scholar
  35. Gerevini, A., & Serina, I. (2002). LPG: A planner based on local search for planning graphs with action costs. In M. Ghallab, J. Hertzberg, & P. Traverso (Eds.) Proceedings of the sixth international conference on artificial intelligence (pp. 13–22). Cambridge: The MIT Press.Google Scholar
  36. Gorissen, D., Couckuyt, I., Demeester, P., Dhaene, T., & Crombecq, K. (2010). A surrogate modeling and adaptive sampling toolbox for computer based design. Journal of Machine Learning Research, 11, 2051–2055.Google Scholar
  37. Guerra, S., Prudêncio, R., & Ludermir, T. (2008). Predicting the performance of learning algorithms using support vector machines as meta-regressors. In V. Kurkova-Pohlova, & J. Koutnik (Eds) International conference on artificial neural networks (ICANN’08) (vol. 18, pp. 523–532). Berlin: Springer.Google Scholar
  38. Hoos, H. (2017). Empirical algorithmics. Cambridge: Cambridge University Press. to appear.Google Scholar
  39. Hoos, H., & Stützle, T. (2004). Stochastic local search: Foundations and applications. Burlington: Morgan Kaufmann Publishers Inc.MATHGoogle Scholar
  40. Hoos, H., Lindauer, M., & Schaub, T. (2014). claspfolio 2: Advances in algorithm selection for answer set programming. Theory and Practice of Logic Programming, 14, 569–585.CrossRefMATHGoogle Scholar
  41. Hutter, F., Babić, D., Hoos, H., & Hu, A. (2007). Boosting verification by automatic tuning of decision procedures. In L. O’Conner (Ed.) Formal methods in computer aided design (FMCAD’07) (pp. 27–34). Washington, DC: IEEE Computer Society Press.Google Scholar
  42. Hutter, F., Hoos, H., Leyton-Brown, K., & Stützle, T. (2009). ParamILS: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36, 267–306.MATHGoogle Scholar
  43. Hutter, F., Hoos, H., & Leyton-Brown, K. (2010). Automated configuration of mixed integer programming solvers. In A. Lodi, M. Milano, & P. Toth (Eds.) Proceedings of the seventh international conference on integration of AI and OR techniques in constraint programming (CPAIOR’10) Lecture Notes in Computer Science (Vol. 6140, pp. 186–202). Berlin: Springer.Google Scholar
  44. Hutter, F., Hoos, H., & Leyton-Brown, K. (2011a). Bayesian optimization with censored response data. In NIPS workshop on Bayesian optimization, sequential experimental design, and bandits (BayesOpt’11).Google Scholar
  45. Hutter, F., Hoos, H., & Leyton-Brown, K, (2011b). Sequential model-based optimization for general algorithm configuration. In C. Coello (Ed.) Proceedings of the fifth international conference on learning and intelligent optimization (LION’11) Lecture Notes in Computer Science (Vol. 6683, pp. 507–523). Belin: Springer.Google Scholar
  46. Hutter, F., López-Ibánez, M., Fawcett, C., Lindauer, M., Hoos, H., Leyton-Brown, K., & Stützle, T. (2014a). Aclib: A benchmark library for algorithm configuration. In P. Pardalos, & M. Resende (Eds.) Proceedings of the eighth international conference on learning and intelligent optimization (LION’14) Lecture Notes in Computer Science. Berlin: Springer.Google Scholar
  47. Hutter, F., Xu, L., Hoos, H., & Leyton-Brown, K. (2014b). Algorithm runtime prediction: Methods and evaluation. Artificial Intelligence, 206, 79–111.MathSciNetCrossRefMATHGoogle Scholar
  48. Hutter, F., Lindauer, M., Balint, A., Bayless, S., Hoos, H., & Leyton-Brown, K. (2017). The configurable SAT solver challenge (CSSC). Artificial Intelligence, 243, 1–25.MathSciNetCrossRefMATHGoogle Scholar
  49. Kadioglu, S., Malitsky, Y., Sellmann, M., & Tierney, K. (2010). ISAC–instance-specific algorithm configuration. In H. Coelho, R. Studer, & M. Wooldridge (Eds.) Proceedings of the nineteenth european conference on artificial intelligence (ECAI’10) (pp. 751–756). Amsterdam: IOS Press.Google Scholar
  50. Koenker, R. (2005). Quantile regression. Cambridge: Cambridge University Press.CrossRefMATHGoogle Scholar
  51. Kotthoff, L. (2014). Algorithm selection for combinatorial search problems: A survey. AI Magazine, 35, 48–60.CrossRefGoogle Scholar
  52. Köpf, C., Taylor, C., & Keller, J. (2000). Meta-analysis: From data characterisation for meta-learning to meta-regression. In Proceedings of the PKDD-00 workshop on data mining, decision support, meta-learning and ILP.Google Scholar
  53. Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). ImageNet classification with deep convolutional neural networks. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, & K. Weinberger (Eds.) Proceedings of the 26th international conference on advances in neural information processing systems (NIPS’12) (pp 1097–1105).Google Scholar
  54. Lang, M., Kotthaus, H., Marwedel, P., Weihs, C., Rahnenführer, J., & Bischl, B. (2015). Automatic model selection for high-dimensional survival analysis. Journal of Statistical Computation and Simulation, 85, 62–76.MathSciNetCrossRefGoogle Scholar
  55. Leite, R., & Brazdil, P. (2010). Active testing strategy to predict the best classification algorithm via sampling and metalearning. In H. Coelho, R. Studer, & M. Wooldridge (Eds.) Proceedings of the nineteenth European conference on artificial intelligence (ECAI’10) (pp. 309–314). Amsterdam: IOS Press.Google Scholar
  56. Leite, R., Brazdil, P., & Vanschoren, J. (2012). Selecting classification algorithms with active testing. In P. Perner (Ed.), Machine learning and data mining in pattern recognition Lecture Notes in Computer Science (pp. 117–131). Berlin: Springer.Google Scholar
  57. Leyton-Brown, K., Pearson, M., & Shoham, Y. (2000). Towards a universal test suite for combinatorial auction algorithms. In Proceedings of the international conference on economics and computation (pp. 66–76).Google Scholar
  58. Leyton-Brown, K., Nudelman, E., & Shoham, Y. (2009). Empirical hardness models: Methodology and a case study on combinatorial auctions. Journal of the ACM, 56(4), 22.MathSciNetCrossRefMATHGoogle Scholar
  59. Lierler, Y., & Schüller, P. (2012). Parsing combinatory categorial grammar via planning in answer set programming. In Lecture Notes in Computer Science (Vol. 7265, pp. 436–453). Berlin: Springer.Google Scholar
  60. Lindauer, M., Hoos, H., Hutter, F., & Schaub, T. (2015). Autofolio: An automatically configured algorithm selector. Journal of Artificial Intelligence Research, 53, 745–778.MathSciNetGoogle Scholar
  61. Long, D., & Fox, M. (2003). The 3rd international planning competition: Results and analysis. Journal of Artificial Intelligence Research (JAIR), 20, 1–59.CrossRefMATHGoogle Scholar
  62. López-Ibáñez, M., Dubois-Lacoste, J., Caceres, L. P., Birattari, M., & Stützle, T. (2016). The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3, 43–58.MathSciNetCrossRefGoogle Scholar
  63. Loreggia, A., Malitsky, Y., Samulowitz, H., & Saraswat, V. (2016). Deep learning for algorithm portfolios. In D. Schuurmans, & M. Wellman (Eds.) Proceedings of the thirtieth national conference on artificial intelligence (AAAI’16) (pp. 1280–1286). AAAI Press.Google Scholar
  64. Malitsky, Y., Sabharwal, A., Samulowitz, H., & Sellmann, M. (2013). Algorithm portfolios based on cost-sensitive hierarchical clustering. In F. Rossi (Ed.) Proceedings of the 23rd international joint conference on artificial intelligence (IJCAI’13) (pp. 608–614).Google Scholar
  65. Manthey, N., & Lindauer, M. (2016). Spybug: Automated bug detection in the configuration space of SAT solvers. In Proceedings of the international conference on theory and applications of satisfiability testing (SAT) (pp. 554–561).Google Scholar
  66. Manthey, N., & Steinke, P. (2014). Too many rooks. In A. Belov, D. Diepold, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2014: Solver and benchmark descriptions (Vol. B-2014-2, pp. 97–98). University of Helsinki, Department of Computer Science Series of Publications B.Google Scholar
  67. Maratea, M., Pulina, L., & Ricca, F. (2014). A multi-engine approach to answer-set programming. Theory and Practice of Logic Programming, 14, 841–868.MathSciNetCrossRefGoogle Scholar
  68. Maron, O., & Moore, A. (1994). Hoeffding races: Accelerating model selection search for classification and function approximation. In Proceedings of the 6th international conference on advances in neural information processing systems (NIPS’94) (pp. 59–66). Burlington: Morgan Kaufmann Publishers.Google Scholar
  69. Meinshausen, N. (2006). Quantile regression forests. Journal of Machine Learning Research, 7, 983–999.MathSciNetMATHGoogle Scholar
  70. Neal, R. (1995). Bayesian learning for neural networks. PhD thesis, University of Toronto, Toronto, Canada.Google Scholar
  71. Nudelman, E., Leyton-Brown, K., Andrew, G., Gomes, C., McFadden, J., Selman, B., & Shoham, Y. (2003). Satzilla 0.9, solver description. International SAT Competition.Google Scholar
  72. Nudelman, E., Leyton-Brown, K., Devkar, A., Shoham, Y., & Hoos, H. (2004). Understanding random SAT: Beyond the clauses-to-variables ratio. In International conference on principles and practice of constraint programming (CP’04) (pp. 438–452).Google Scholar
  73. Oh, C. (2014). Minisat hack 999ed, minisat hack 1430ed and swdia5by. In A. Belov, D. Diepold, M. Heule, & M. Järvisalo (Eds.) Proceedings of SAT competition 2014: solver and benchmark descriptions (Vol. B-2014-2, p. 46). University of Helsinki, Department of Computer Science Series of Publications B.Google Scholar
  74. Penberthy, J., & Weld, D. (1994). Temporal planning with continuous change. In B. Hayes-Roth, & R. Korf (Eds.) Proceedings of the 12th national conference on artificial intelligence (pp. 1010–1015). Cambridge: The MIT Press.Google Scholar
  75. Rasmussen, C., & Williams, C. (2006). Gaussian processes for machine learning. Cambridge: The MIT Press.MATHGoogle Scholar
  76. Reif, M., Shafait, F., Goldstein, M., Breuel, T., & Dengel, A. (2014). Automatic classifier selection for non-experts. Pattern Analysis and Applications, 17(1), 83–96.MathSciNetCrossRefGoogle Scholar
  77. Rice, J. (1976). The algorithm selection problem. Advances in Computers, 15, 65–118.CrossRefGoogle Scholar
  78. Sacks, J., Welch, W., Welch, T., & Wynn, H. (1989). Design and analysis of computer experiments. Statistical Science, 4(4), 409–423.MathSciNetCrossRefMATHGoogle Scholar
  79. Santner, T., Williams, B., & Notz, W. (2003). The design and analysis of computer experiments. Berlin: Springer.CrossRefMATHGoogle Scholar
  80. Sarkar, A., Guo, J., Siegmund, N., Apel, S., & Czarnecki, K. (2015). Cost-efficient sampling for performance prediction of configurable systems. In M. Cohen, L. Grunske, & M. Whalen (Eds.) 30th IEEE/ACM International Conference on Automated Software Engineering (pp. 342–352). IEEE.Google Scholar
  81. Schilling, N., Wistuba, M., Drumond, L., & Schmidt-Thieme, L. (2015). Hyperparameter optimization with factorized multilayer perceptrons. In Machine learning and knowledge discovery in databases (pp. 87–103). Berlin: Springer.Google Scholar
  82. Schmee, J., & Hahn, G. (1979). A simple method for regression analysis with censored data. Technometrics, 21, 417–432.CrossRefGoogle Scholar
  83. Shahriari, B., Swersky, K., Wang, Z., Adams, R., & de Freitas, N. (2016). Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1), 148–175.CrossRefGoogle Scholar
  84. Silverthorn, B., Lierler, Y., & Schneider, M. (2012). Surviving solver sensitivity: An ASP practitioner’s guide. In A. Dovier, & V. Santos Costa (Eds.) Technical communications of the twenty-eighth international conference on logic programming (ICLP’12), Leibniz international proceedings in informatics (LIPIcs) (Vol. 17, pp. 164–175).Google Scholar
  85. Snoek, J., Larochelle, H., & Adams, RP. (2012). Practical Bayesian optimization of machine learning algorithms. In P. Bartlett, F. Pereira, C. Burges, L. Bottou, & K. Weinberger (Eds.) Proceedings of the 26th international conference on advances in neural information processing systems (NIPS’12) (pp. 2960–2968).Google Scholar
  86. Snoek J, Rippel O, Swersky K, Kiros R, Satish N, Sundaram N, Patwary M, Prabhat, Adams R (2015). Scalable Bayesian optimization using deep neural networks. In F. Bach, & D. Blei (Eds.) Proceedings of the 32nd international conference on machine learning (ICML’15) (Vol. 37, pp. 2171–2180). Madison: Omnipress.Google Scholar
  87. Soares, C., & Brazdil, P. (2004). A meta-learning method to select the kernel width in support vector regression. Machine Learning Journal, 54, 195–209.CrossRefMATHGoogle Scholar
  88. Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 71–101.Google Scholar
  89. Springenberg, J., Klein, A., Falkner, S., & Hutter, F. (2016). Bayesian optimization with robust Bayesian neural networks. In Proceedings of the international conference on advances in neural information processing systems (NIPS’16).Google Scholar
  90. Takeuchi, I., Le, Q., Sears, T., & Smola, A. (2006). Nonparametric quantile estimation. Journal of Machine Learning Research, 7, 1231–1264.MathSciNetMATHGoogle Scholar
  91. Thornton, C., Hutter, F., Hoos, H., & Leyton-Brown, K. (2013). Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In I. Dhillon, Y. Koren, R. Ghani, T. Senator, P. Bradley, R. Parekh, J. He, R. Grossman, & R. Uthurusamy (Eds.) The 19th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’13) (pp 847–855). New York: ACM PressGoogle Scholar
  92. Vallati, M., Fawcett, C., Gerevini, A., Hoos, H., & Saetti, A. (2013). Automatic generation of efficient domain-optimized planners from generic parametrized planners. In M. Helmert, & G. Röger (Eds.) Proceedings of the sixth annual symposium on combinatorial search (SOCS’14), AAAI Press.Google Scholar
  93. Wistuba, M., Schilling, N., & Schmidt-Thieme, L. (2015). Learning hyperparameter optimization initializations. In Proceedings of the international conference on data science and advanced analytics (DSAA) (pp. 1–10). IEEEGoogle Scholar
  94. Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2008). SATzilla: Portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research, 32, 565–606.MATHGoogle Scholar
  95. Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2011). Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming. In RCRA workshop on Experimental Evaluation of Algorithms for Solving Problems with Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI).Google Scholar

Copyright information

© The Author(s) 2017

Authors and Affiliations

  1. 1.University of FreiburgFreiburg im BreisgauGermany
  2. 2.University of British ColumbiaVancouverCanada

Personalised recommendations