Summary
This chapter discusses different types of metalearning models, including regression, classification and relative performance models. Regression models use a suitable regression algorithm, which is trained on the metadata and used to predict the performance of given base-level algorithms. The predictions can in turn be used to order the base-level algorithms and hence identify the best one. These models also play an important role in the search for the potentially best hyperparameter configuration discussed in the next chapter. Classification models identify which base-level algorithms are applicable or non-applicable to the target classification task. Probabilistic classifiers can be used to construct a ranking of potentially useful alternatives. Relative performance models exploit information regarding the relative performance of base-level models, which can be either in the form of rankings or pairwise comparisons. This chapter discusses various methods that use this information in the search for the potentially best algorithm for the target task.
Download chapter PDF
References
Abdulrahman, S., Brazdil, P., van Rijn, J. N., and Vanschoren, J. (2018). Speeding up algorithm selection using average ranking and active testing by introducing runtime. Machine Learning, 107(1):79–108.
Bensusan, H. and Giraud-Carrier, C. (2000). Discovering task neighbourhoods through landmark learning performances. In Zighed, D. A., Komorowski, J., and Zytkow, J., editors, Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD 2000), pages 325–330. Springer.
Bensusan, H. and Kalousis, A. (2001). Estimating the predictive accuracy of a classifier. In Flach, P. and De Raedt, L., editors, Proceedings of the 12th European Conference on Machine Learning, pages 25–36. Springer.
Blockeel, H., De Raedt, L., and Ramon, J. (1998). Top-down induction of clustering trees. In Proceedings of the 15th International Conference on Machine Learning, ICML’98, pages 55–63, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Brazdil, P., Gama, J., and Henery, B. (1994). Characterizing the applicability of classification algorithms using meta-level learning. In Bergadano, F. and De Raedt, L., editors, Proceedings of the European Conference on Machine Learning (ECML94), pages 83–102. Springer-Verlag.
Brazdil, P., Soares, C., and da Costa, J. P. (2003). Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3):251–277.
Cohen,W.W. (1995). Fast effective rule induction. In Prieditis, A. and Russell, S., editors, Proceedings of the 12th International Conference on Machine Learning, ICML’95, pages 115–123. Morgan Kaufmann.
Dimitriadou, E., Hornik, K., Leisch, F., Meyer, D., andWeingessel, A. (2004). e1071: Misc functions of the Department of Statistics (e1071), R package version 1.5-1. Technical report, TU Wien.
Domhan, T., Springenberg, J. T., and Hutter, F. (2015). Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Eggensperger, K., Lindauer, M., Hoos, H., Hutter, F., and Leyton-Brown, K. (2018). Efficient benchmarking of algorithm configuration procedures via model-based surrogates. Special Issue on Metalearning and Algorithm Selection, Machine Learning, 107(1).
Ferreira, M. and Brazdil, P. (2018). Workflow recommendation for text classification with active testing method. In Workshop AutoML 2018 @ ICML/IJCAI-ECAI. Available at site https://sites.google.com/site/automl2018icml/accepted-papers.
Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F. (2018). Practical automated machine learning for the AutoML challenge 2018. In International Workshop on Automatic Machine Learning at ICML2018, pages 1189–1232.
Feurer, M., Springenberg, J. T., and Hutter, F. (2014). Using meta-learning to initialize Bayesian optimization of hyperparameters. In ECAI Workshop on Metalearning and Algorithm Selection (MetaSel), pages 3–10.
F¨urnkranz, J. and Petrak, J. (2001). An evaluation of landmarking variants. In Giraud-Carrier, C., Lavrač, N., and Moyle, S., editors, Working Notes of the ECML/PKDD 2000 Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning, pages 57–68.
Gama, J. and Brazdil, P. (1995). Characterization of classification algorithms. In Pinto-Ferreira, C. and Mamede, N. J., editors, Progress in Artificial Intelligence, Proceedings of the Seventh Portuguese Conference on Artificial Intelligence, pages 189–200. Springer-Verlag.
Hilario, M. and Kalousis, A. (2001). Fusion of meta-knowledge and meta-data for casebased model selection. In Siebes, A. and De Raedt, L., editors, Proceedings of the Fifth European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD01). Springer.
Hutter, F., Xu, L., Hoos, H., and Leyton-Brown, K. (2014). Algorithm runtime prediction: Methods and evaluation. Artificial Intelligence, 206:79–111.
Jamieson, K. and Talwalkar, A. (2016). Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pages 240–248.
Kalousis, A. (2002). Algorithm Selection via Meta-Learning. PhD thesis, University of Geneva, Department of Computer Science.
Kalousis, A. and Hilario, M. (2000). Model selection via meta-learning: A comparative study. In Proceedings of the 12th International IEEE Conference on Tools with AI. IEEE Press.
Kalousis, A. and Hilario, M. (2003). Representational issues in meta-learning. In Proceedings of the 20th International Conference on Machine Learning, ICML’03, pages 313–320.
Kolodner, J. (1993). Case-Based Reasoning. Morgan Kaufmann Publishers.
K¨opf, C., Taylor, C., and Keller, J. (2000). Meta-analysis: From data characterization for meta-learning to meta-regression. In Brazdil, P. and Jorge, A., editors, Proceedings of the PKDD 2000 Workshop on Data Mining, Decision Support, Meta-Learning and ILP: Forum for Practical Problem Presentation and Prospective Solutions, pages 15–26.
Leake, D. B. (1996). Case-Based Reasoning: Experiences, Lessons & Future Directions. AAAI Press.
Leite, R. and Brazdil, P. (2004). Improving progressive sampling via meta-learning on learning curves. In Boulicaut, J.-F., Esposito, F., Giannotti, F., and Pedreschi, D., editors, Proc. of the 15th European Conf. on Machine Learning (ECML2004), LNAI 3201, pages 250–261. Springer-Verlag.
Leite, R. and Brazdil, P. (2005). Predicting relative performance of classifiers from samples. In Proceedings of the 22nd International Conference on Machine Learning, ICML’05, pages 497–503, NY, USA. ACM Press.
Leite, R. and Brazdil, P. (2007). An iterative process for building learning curves and predicting relative performance of classifiers. In Proceedings of the 13th Portuguese Conference on Artificial Intelligence (EPIA 2007), pages 87–98.
Leite, R. and Brazdil, P. (2010). Active testing strategy to predict the best classification algorithm via sampling and metalearning. In Proceedings of the 19th European Conference on Artificial Intelligence (ECAI), pages 309–314.
Leite, R. and Brazdil, P. (2021). Exploiting performance-based similarity between datasets in metalearning. In Guyon, I., van Rijn, J. N., Treguer, S., and Vanschoren, J., editors, AAAI Workshop on Meta-Learning and MetaDL Challenge, volume 140, pages 90–99. PMLR.
Leite, R., Brazdil, P., and Vanschoren, J. (2012). Selecting classification algorithms with active testing. In Machine Learning and Data Mining in Pattern Recognition, pages 117–131. Springer.
Leyton-Brown, K., Nudelman, E., and Shoham, Y. (2009). Empirical hardness models: Methodology and a case study on combinatorial auctions. Journal of the ACM, 56(4).
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2017). Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization. In Proc. of ICLR 2017.
Lindner, G. and Studer, R. (1999). AST: Support for algorithm selection with a CBR approach. In Giraud-Carrier, C. and Pfahringer, B., editors, Recent Advances in Meta-Learning and Future Work, pages 38–47. J. Stefan Institute.
Mohr, F. and van Rijn, J. N. (2021). Towards model selection using learning curve crossvalidation. In 8th ICML Workshop on Automated Machine Learning (AutoML).
Pfahringer, B., Bensusan, H., and Giraud-Carrier, C. (2000). Meta-learning by landmarking various learning algorithms. In Langley, P., editor, Proceedings of the 17th International Conference on Machine Learning, ICML’00, pages 743–750.
Pfisterer, F., van Rijn, J. N., Probst, P., M¨uller, A., and Bischl, B. (2018). Learning multiple defaults for machine learning algorithms. arXiv preprint arXiv:1811.09409.
Provost, F., Jensen, D., and Oates, T. (1999). Efficient progressive sampling. In Chaudhuri, S. and Madigan, D., editors, Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM.
Quinlan, J. (1992). Learning with continuous classes. In Adams and Sterling, editors, AI’92, pages 343–348. Singapore: World Scientific.
Quinlan, R. (1998). C5.0: An Informal Tutorial. RuleQuest. http://www.rulequest.com/see5-unix.html.
Quinlan, R. and Cameron-Jones, R. (1993). FOIL: A midterm report. In Brazdil, P., editor, Proc. of the Sixth European Conf. on Machine Learning, volume 667 of LNAI, pages 3–20. Springer-Verlag.
Rasmussen, C. and Williams, C. (2006). Gaussian Processes for Machine Learning. The MIT Press.
Soares, C., Petrak, J., and Brazdil, P. (2001). Sampling-based relative landmarks: Systematically test-driving algorithms before choosing. In Brazdil, P. and Jorge, A., editors, Proceedings of the 10th Portuguese Conference on Artificial Intelligence (EPIA2001), pages 88–94. Springer.
Sohn, S. Y. (1999). Meta analysis of classification algorithms for pattern recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(11):1137–1144.
Sun, Q. and Pfahringer, B. (2012). Bagging ensemble selection for regression. In Proceedings of the 25th Australasian Joint Conference on Artificial Intelligence, pages 695–706.
Sun, Q. and Pfahringer, B. (2013). Pairwise meta-rules for better meta-learning-based algorithm ranking. Machine Learning, 93(1):141–161.
Thornton, C., Hutter, F., Hoos, H. H., and Leyton-Brown, K. (2013). Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 847–855. ACM.
Todorovski, L., Blockeel, H., and Džeroski, S. (2002). Ranking with predictive clustering trees. In Elomaa, T., Mannila, H., and Toivonen, H., editors, Proc. of the 13th European Conf. on Machine Learning, number 2430 in LNAI, pages 444–455. Springer-Verlag.
Todorovski, L. and Džeroski, S. (1999). Experiments in meta-level learning with ILP. In Rauch, J. and Zytkow, J., editors, Proceedings of the Third European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD99), pages 98–106.Springer.
Tsuda, K., Rätsch, G., Mika, S., and Müller, K. (2001). Learning to predict the leave-oneout error of kernel based classifiers. In ICANN, pages 331–338. Springer-Verlag.
van Rijn, J. N. (2016). Massively collaborative machine learning. PhD thesis, Leiden University.
van Rijn, J. N., Abdulrahman, S., Brazdil, P., and Vanschoren, J. (2015). Fast algorithm selection using learning curves. In International Symposium on Intelligent Data Analysis XIV, pages 298–309.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2022 The Author(s)
About this chapter
Cite this chapter
Brazdil, P., van Rijn, J.N., Soares, C., Vanschoren, J. (2022). Metalearning Approaches for Algorithm Selection II. In: Metalearning. Cognitive Technologies. Springer, Cham. https://doi.org/10.1007/978-3-030-67024-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-67024-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67023-8
Online ISBN: 978-3-030-67024-5
eBook Packages: Computer ScienceComputer Science (R0)