Abstract
Per-instance automated algorithm configuration and selection are gaining significant moments in evolutionary computation in recent years. Two crucial, sometimes implicit, ingredients for these automated machine learning (AutoML) methods are 1) feature-based representations of the problem instances and 2) performance prediction methods that take the features as input to estimate how well a specific algorithm instance will perform on a given problem instance. Non-surprisingly, common machine learning models fail to make predictions for instances whose feature-based representation is underrepresented or not covered in the training data, resulting in poor generalization ability of the models for problems not seen during training. In this work, we study leave-one-problem-out (LOPO) performance prediction. We analyze whether standard random forest (RF) model predictions can be improved by calibrating them with a weighted average of performance values obtained by the algorithm on problem instances that are sufficiently similar to the problem for which a performance prediction is sought, measured by cosine similarity in feature space. While our RF+clust approach obtains more accurate performance prediction for several problems, its predictive power crucially depends on the chosen similarity threshold as well as on the feature portfolio for which the cosine similarity is measured, thereby opening a new angle for feature selection in a zero-shot learning setting, as LOPO is termed in machine learning.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Belkhir, N., Dréo, J., Savéant, P., Schoenauer, M.: Per instance algorithm configuration of CMA-ES with limited budget. In: GECCO, pp. 681–688 (2017)
Biau, G., Scornet, E.: A random forest guided tour. TEST 25(2), 197–227 (2016). https://doi.org/10.1007/s11749-016-0481-7
Bischl, B., Mersmann, O., Trautmann, H., Preuß, M.: Algorithm selection based on exploratory landscape analysis and cost-sensitive learning. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 313–320 (2012)
Cenikj, G., Lang, R.D., Engelbrecht, A.P., Doerr, C., Korošec, P., Eftimov, T.: Selector: selecting a representative benchmark suite for reproducible statistical comparison. arXiv preprint arXiv:2204.11527 (2022)
Derbel, B., Liefooghe, A., Vérel, S., Aguirre, H., Tanaka, K.: New features for continuous exploratory landscape analysis based on the soo tree. In: FOGA, pp. 72–86 (2019)
Dietrich, K., Mersmann, O.: Increasing the diversity of benchmark function sets through affine recombination. In: Rudolph, G., Kononova, A.V., Aguirre, H., Kerschke, P., Ochoa, G., Tušar, T. (eds.) International Conference on Parallel Problem Solving from Nature, pp. 590–602. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-14714-2_41
Eftimov, T., Jankovic, A., Popovski, G., Doerr, C., Korošec, P.: Personalizing performance regression models to black-box optimization problems. In: GECCO, pp. 669–677 (2021)
Eftimov, T., et al.: Less is more: Selecting the right benchmarking set of data for time series classification. Expert Syst. Appl. 198, 116871 (2022)
Hansen, N., Auger, A., Finck, S., Ros, R.: Real-parameter black-box optimization benchmarking 2010: Experimental setup. Ph.D. thesis, INRIA (2010)
Hansen, N., Auger, A., Ros, R., Mersmann, O., Tušar, T., Brockhoff, D.: COCO: a platform for comparing continuous optimizers in a black-box setting. Optim. Meth. Software 36(1), 114–144 (2021)
Jankovic, A., Doerr, C.: Landscape-aware fixed-budget performance regression and algorithm selection for modular CMA-ES variants. In: GECCO, pp. 841–849. ACM (2020)
Jankovic, A., Popovski, G., Eftimov, T., Doerr, C.: The impact of hyper-parameter tuning for landscape-aware performance regression and algorithm selection. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 687–696 (2021)
Kerschke, P., Trautmann, H.: The r-package flacco for exploratory landscape analysis with applications to multi-objective optimization problems. In: 2016 IEEE Congress on Evolutionary Computation (CEC), pp. 5262–5269. IEEE (2016)
Kostovska, A., et al.: Per-run algorithm selection with warm-starting using trajectory-based features. arXiv preprint arXiv:2204.09483 (2022)
Lang, R.D., Engelbrecht, A.P.: An exploratory landscape analysis-based benchmark suite. Algorithms 14(3), 78 (2021)
Liang, J.J., Qu, B.Y., Suganthan, P.N.: Problem definitions and evaluation criteria for the cec 2014 special session and competition on single objective real-parameter numerical optimization. Technical report Zhengzhou, China 635, 490 (2013)
Malan, K.M., Engelbrecht, A.P.: Fitness landscape analysis for metaheuristic performance prediction. In: Richter, H., Engelbrecht, A. (eds.) Recent Advances in the Theory and Application of Fitness Landscapes. ECC, vol. 6, pp. 103–132. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-41888-4_4
Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., Rudolph, G.: Exploratory landscape analysis. In: GECCO, pp. 829–836 (2011)
Molnar, C.: Interpretable machine learning. Lulu. com (2020)
Nikolikj, A.: Rfclustgit (2023). https://github.com/anikolik/RF-clust
Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Petelin, G., Cenikj, G., Eftimov, T.: Tla: Topological landscape analysis for single objective continuous optimization problem instances. In: 2022 IEEE Symposium Series on Computational Intelligence (SSCI). p. In Press. IEEE (2022)
Rapin, J., Teytaud, O.: Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad (2018)
Škvorc, U., Eftimov, T., Korošec, P.: Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Appl. Soft Comput. 90, 106138 (2020)
Škvorc, U., Eftimov, T., Korošec, P.: Transfer learning analysis of multi-class classification for landscape-aware algorithm selection. Mathematics 10(3), 432 (2022)
Stork, J., Eiben, A.E., Bartz-Beielstein, T.: A new taxonomy of global optimization algorithms. Natural Comput. 21, 219–242 (2020). https://doi.org/10.1007/s11047-020-09820-4
Storn, R., Price, K.: Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J. Global Optim. 11(4), 341–359 (1997)
Xu, Q., Yang, Y., Liu, Y., Wang, X.: An improved Latin hypercube sampling method to enhance numerical stability considering the correlation of input variables. IEEE Access 5, 15197–15205 (2017)
Acknowledgments
The authors acknowledge the support of the Slovenian Research Agency through program grant P2-0098, project grants N2-0239 and J2-4460, and a bilateral project between Slovenia and France grant No. BI-FR/23-24-PROTEUS-001 (PR-12040). Our work is also supported by ANR-22-ERCS-0003-01 project VARIATION.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nikolikj, A., Doerr, C., Eftimov, T. (2023). RF+clust for Leave-One-Problem-Out Performance Prediction. In: Correia, J., Smith, S., Qaddoura, R. (eds) Applications of Evolutionary Computation. EvoApplications 2023. Lecture Notes in Computer Science, vol 13989. Springer, Cham. https://doi.org/10.1007/978-3-031-30229-9_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-30229-9_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30228-2
Online ISBN: 978-3-031-30229-9
eBook Packages: Computer ScienceComputer Science (R0)