Machine Learning

, Volume 107, Issue 1, pp 79–108 | Cite as

Speeding up algorithm selection using average ranking and active testing by introducing runtime

  • Salisu Mamman Abdulrahman
  • Pavel Brazdil
  • Jan N. van Rijn
  • Joaquin Vanschoren
Part of the following topical collections:
  1. Special Issue on Metalearning and Algorithm Selection


Algorithm selection methods can be speeded-up substantially by incorporating multi-objective measures that give preference to algorithms that are both promising and fast to evaluate. In this paper, we introduce such a measure, A3R, and incorporate it into two algorithm selection techniques: average ranking and active testing. Average ranking combines algorithm rankings observed on prior datasets to identify the best algorithms for a new dataset. The aim of the second method is to iteratively select algorithms to be tested on the new dataset, learning from each new evaluation to intelligently select the next best candidate. We show how both methods can be upgraded to incorporate a multi-objective measure A3R that combines accuracy and runtime. It is necessary to establish the correct balance between accuracy and runtime, as otherwise time will be wasted by conducting less informative tests. The correct balance can be set by an appropriate parameter setting within function A3R that trades off accuracy and runtime. Our results demonstrate that the upgraded versions of Average Ranking and Active Testing lead to much better mean interval loss values than their accuracy-based counterparts.


Algorithm selection Meta-learning Ranking of algorithms Average ranking Active testing Loss curves Mean interval loss 



The authors wish to express our gratitude to the following institutions which have provided funding to support this work: Federal Government of Nigeria Tertiary Education Trust Fund under the TETFund 2012 AST$D Intervention for Kano University of Science and Technology, Wudil, Kano State, Nigeria for PhD Overseas Training; FCT/MEC through PIDDAC and ERDF/ON2 within project NORTE-07-0124-FEDER-000059 and through the COMPETE Programme (operational programme for competitiveness) and by National Funds through the FCT Portuguese Foundation for Science and Technology within project FCOMP-01-0124-FEDER-037281; Grant 612:001:206 from the Netherlands Organisation for Scientific Research (NWO). We wish to express our gratitude to all anonymous referees for their detailed comments which led to various improvements of this paper. Also, our thanks to Miguel Cachada for reading through the paper and his comments and in particular one very useful observation regarding the AT method.


  1. Abdulrahman, S. M., & Brazdil, P. (2014). Measures for combining accuracy and time for meta-learning. Meta-Learning and Algorithm Selection Workshop at ECAI, 2014, 49–50.Google Scholar
  2. Brazdil, P. & Soares, C. (2000). A Comparison of ranking methods for classification algorithm selection. In Machine Learning: ECML 2000, pp. 63–75. Springer.Google Scholar
  3. Brazdil, P., Soares, C., & da Costa, J. P. (2003). Ranking learning algorithms: Using IBL and meta-learning on accuracy and time results. Machine Learning, 50(3), 251–277.CrossRefzbMATHGoogle Scholar
  4. Brazdil, P., Carrier, C . G., Soares, C., & Vilalta, R. (2008). Metalearning: Applications to data mining. Berlin: Springer.zbMATHGoogle Scholar
  5. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1–30.MathSciNetzbMATHGoogle Scholar
  6. Fedorov, V . V. (1972). Theory of optimal experiments. Cambridge: Academic Press.Google Scholar
  7. Feurer, M., Springenberg, T. & Hutter, F. (2015). Initializing bayesian hyperparameter optimization via meta-learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 2015.Google Scholar
  8. Fürnkranz, J. & Petrak, J. (2001). An evaluation of landmarking variants. In Working Notes of the ECML/PKDD 2000 Workshop on Integrating Aspects of Data Mining, Decision Support and Meta-Learning, pp. 57–68.Google Scholar
  9. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, 11(1), 10–18.CrossRefGoogle Scholar
  10. Jankowski, N. (2013). Complexity measures for meta-learning and their optimality. In Algorithmic Probability and Friends. Bayesian Prediction and Artificial Intelligence., pp. 198–210.Google Scholar
  11. Leite, R. & Brazdil, P. (2008). Selecting classifiers using meta-learning with sampling landmarks and data characterization. In Proceedings of the 2nd Planning to Learn Workshop (PlanLearn) at ICML/COLT/UAI 2008, pp. 35–41.Google Scholar
  12. Leite, R. & Brazdil, P. (2010). Active testing strategy to predict the best classification algorithm via sampling and metalearning. In ECAI, pp. 309–314.Google Scholar
  13. Leite, R., Brazdil, P. & Vanschoren, J. (2012). Selecting classification algorithms with active testing. In Machine Learning and Data Mining in Pattern Recognition, pp. 117–131. Springer.Google Scholar
  14. Li, L., Chu, W., Langford, J. & Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of 19th WWW, pp. 661–670. ACM.Google Scholar
  15. Lin, S. (2010). Rank aggregation methods. WIREs Computational Statistics, 2, 555–570.CrossRefGoogle Scholar
  16. Long, B., Chapelle, O., Zhang, Y., Chang, Y., Zheng, Z. & Tseng, B. (2010). Active learning for ranking through expected loss optimization. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 267–274. ACM.Google Scholar
  17. Neave, H. R., & Worthington, P. L. (1988). Distribution-free Tests. London: Unwin Hyman.Google Scholar
  18. Pfahringer, B., Bensusan, H. & Giraud-Carrier, C. (2000). Tell me who can learn you and I can tell you who you are: Landmarking various learning algorithms. In Proceedings of the 17th International Conference on Machine Learning, pp. 743–750.Google Scholar
  19. Pihur, V., Datta, S., & Datta, S. (2009). RankAggreg, an R package for weighted rank aggregation. BMC Bioinformatics, 10(1), 62.CrossRefGoogle Scholar
  20. Prudêncio, R. B. & Ludermir, T. B. (2007). Active selection of training examples for meta-learning. In 7th International Conference on Hybrid Intelligent Systems, 2007. HIS 2007., pp. 126–131. IEEE.Google Scholar
  21. Rice, J. R. (1976). The algorithm selection problem. Advances in Computers, 15, 65–118.CrossRefGoogle Scholar
  22. Smith-Miles, K . A. (2008). Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Computing Surveys (CSUR), 41(1), 6:1–6:25.CrossRefGoogle Scholar
  23. Sun, Quan, & Pfahringer, B. (2013). Pairwise meta-rules for better meta-learning-based algorithm ranking. Machine Learning, 93(1), 141–161.MathSciNetCrossRefzbMATHGoogle Scholar
  24. van Rijn, J. N., Abdulrahman, S. M., Brazdil, P. & Vanschoren, J. (2015). Fast algorithm selection using learning curves. In Advances in Intelligent Data Analysis XIV, pp. 298–309. Springer.Google Scholar
  25. Vanschoren, J., van Rijn, J. N., Bischl, B., & Torgo, L. (2014). OpenML: Networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2), 49–60.CrossRefGoogle Scholar
  26. Witten, I . H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques. Burlington: Morgan Kaufmann.zbMATHGoogle Scholar
  27. Xu, L., Hutter, F., Hoos, H., & Leyton-Brown, K. (2011). Hydra-MIP: Automated algorithm configuration and selection for mixed integer programming. In RCRA workshop on Experimental Evaluation of Algorithms for Solving Problems with Combinatorial Explosion at the International Joint Conference on Artificial Intelligence (IJCAI)Google Scholar

Copyright information

© The Author(s) 2017

Authors and Affiliations

  • Salisu Mamman Abdulrahman
    • 1
    • 2
  • Pavel Brazdil
    • 2
    • 3
  • Jan N. van Rijn
    • 4
    • 5
  • Joaquin Vanschoren
    • 6
  1. 1.Kano University of Science and TechnologyWudilNigeria
  2. 2.LIAADINESC TEC, Rua Dr. Roberto FriasPortoPortugal
  3. 3.Faculdade de EconomiaUniversidade do Porto, Rua Dr. Roberto FriasPortoPortugal
  4. 4.University of FreiburgFreiburgGermany
  5. 5.Leiden Institute of Advanced Computer ScienceLeiden UniversityLeidenThe Netherlands
  6. 6.Eindhoven University of TechnologyEindhovenNetherlands

Personalised recommendations