Fast Algorithm Selection Using Learning Curves

  • Jan N. van RijnEmail author
  • Salisu Mamman Abdulrahman
  • Pavel Brazdil
  • Joaquin Vanschoren
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9385)


One of the challenges in Machine Learning to find a classifier and parameter settings that work well on a given dataset. Evaluating all possible combinations typically takes too much time, hence many solutions have been proposed that attempt to predict which classifiers are most promising to try. As the first recommended classifier is not always the correct choice, multiple recommendations should be made, making this a ranking problem rather than a classification problem. Even though this is a well studied problem, there is currently no good way of evaluating such rankings. We advocate the use of Loss Time Curves, as used in the optimization literature. These visualize the amount of budget (time) needed to converge to a acceptable solution. We also investigate a method that utilizes the measured performances of classifiers on small samples of data to make such recommendation, and adapt it so that it works well in Loss Time space. Experimental results show that this method converges extremely fast to an acceptable solution.


Algorithm selection Meta-learning Subsampling 



This work is supported by grant \(600.065.120.12\mathrm {N}150\) from the Dutch Fund for Scientific Research (NWO).


  1. 1.
    Abdulrahman, S.M., Brazdil, P.: Measures for combining accuracy and time for meta-learning. In: Meta-Learning and Algorithm Selection Workshop at ECAI, 2014, pp. 49–50 (2014)Google Scholar
  2. 2.
    Brazdil, P., Gama, J., Henery, B.: Characterizing the applicability of classification algorithms using meta-level learning. In: Bergadano, F., De Raedt, L. (eds.) ECML-94. LNCS, vol. 784, pp. 83–102. Springer, Heidelberg (1994)CrossRefGoogle Scholar
  3. 3.
    Brazdil, P.B., Soares, C.: A comparison of ranking methods for classification algorithm selection. In: Lopez de Mantaras, R., Plaza, E. (eds.) ECML 2000. LNCS (LNAI), vol. 1810, pp. 63–74. Springer, Heidelberg (2000) CrossRefGoogle Scholar
  4. 4.
    Fürnkranz, J., Petrak, J.: An evaluation of landmarking variants. In: Working Notes of the ECML/PKDD 2000 Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning, pp. 57–68 (2001)Google Scholar
  5. 5.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. ACM SIGKDD Explor. Newsl. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  6. 6.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Murphy, K.: Time-bounded sequential parameter optimization. In: Blum, C., Battiti, R. (eds.) LION 4. LNCS, vol. 6073, pp. 281–298. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  7. 7.
    Leite, R., Brazdil, P.: Predicting relative performance of classifiers from samples. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 497–503. ACM (2005)Google Scholar
  8. 8.
    Leite, R., Brazdil, P.: Active testing strategy to predict the best classification algorithm via sampling and metalearning. In: ECAI, pp. 309–314 (2010)Google Scholar
  9. 9.
    Leite, R., Brazdil, P., Vanschoren, J.: Selecting classification algorithms with active testing. In: Perner, P. (ed.) MLDM 2012. LNCS, vol. 7376, pp. 117–131. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  10. 10.
    Petrak, J.: Fast subsampling performance estimates for classification algorithm selection. In: Proceedings of the ECML-00 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pp. 3–14 (2000)Google Scholar
  11. 11.
    Pfahringer, B., Bensusan, H., Giraud-Carrier, C.: Tell me who can learn you and i can tell you who you are: Landmarking various learning algorithms. In: Proceedings of the 17th International Conference on Machine Learning, pp. 743–750 (2000)Google Scholar
  12. 12.
    Provost, F., Jensen, D., Oates, T.: Efficient progressive sampling. In: Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 23–32. ACM (1999)Google Scholar
  13. 13.
    Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65118 (1976)Google Scholar
  14. 14.
    van Rijn, J.N., Holmes, G., Pfahringer, B., Vanschoren, J.: Algorithm selection on data streams. In: Džeroski, S., Panov, P., Kocev, D., Todorovski, L. (eds.) DS 2014. LNCS, vol. 8777, pp. 325–336. Springer, Heidelberg (2014) Google Scholar
  15. 15.
    Rossi, A.L.D., de Leon Ferreira, A.C.P., Soares, C., De Souza, B.F.: MetaStream: a meta-learning based method for periodic algorithm selection in time-changing data. Neurocomputing 127, 52–64 (2014)CrossRefGoogle Scholar
  16. 16.
    Smith-Miles, K.A.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. (CSUR) 41(1), 6 (2008)CrossRefGoogle Scholar
  17. 17.
    Sun, Q., Pfahringer, B.: Pairwise meta-rules for better meta-learning-based algorithm ranking. Mach. Learn. 93(1), 141–161 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Vanschoren, J., Blockeel, H., Pfahringer, B., Holmes, G.: Experiment databases. Mach. Learn. 87(2), 127–158 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. ACM SIGKDD Explor. Newsl. 15(2), 49–60 (2014)CrossRefGoogle Scholar
  20. 20.
    Vilalta, R., Drissi, Y.: A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2), 77–95 (2002)CrossRefGoogle Scholar
  21. 21.
    Wolpert, D.H.: Stacked generalization. Neural Networks 5(2), 241–259 (1992)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Jan N. van Rijn
    • 1
    Email author
  • Salisu Mamman Abdulrahman
    • 2
  • Pavel Brazdil
    • 2
  • Joaquin Vanschoren
    • 3
  1. 1.Leiden UniversityLeidenThe Netherlands
  2. 2.University of PortoPortoPortugal
  3. 3.Eindhoven University of TechnologyEindhovenThe Netherlands

Personalised recommendations