Racing with a Fixed Budget and a Self-Adaptive Significance Level

  • Juergen BrankeEmail author
  • Jawad Elomari
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7997)


F-Race is an offline parameter tuning method which efficiently allocates samples in order to identify the parameter setting with the best expected performance, out of a given set of parameter settings. Using non parametric statistical tests, F-Race discards parameter settings which perform significantly worse than the current best, allowing the surviving parameter settings to be tested on more instances and hence obtaining better estimates for their performance. The statistical tests require setting significance levels which directly affect the algorithm’s ability of detecting the best parameter setting, and the total runtime. In this paper, we show that it is not straightforward to set the significance level and propose a simple modification to automatically adapt the significance level such that the failure rate is minimized. This is tested empirically using data drawn from probability distributions with pre-defined characteristics. Results indicate that, under a strict computational budget, F-Race with online adaptation performs significantly better than its counterpart with even the best fixed value.


Quadratic Assignment Problem Timetabling Problem Covariance Matrix Adaptation Evolution Strategy Equal Allocation Good Parameter Setting 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Maron, O., Moore, A.: Hoeffding races: accelerating model selection search for classification and function approximation. Adv. Neural Inf. Process. Syst. 6, 59–66 (1994)Google Scholar
  2. 2.
    Schaffer, J.D., Caruana, R.A., Eshelman, L.J., Das, R.: A study of control parameters affecting online performance of genetic algorithms for function optimization. In: International Conference on Genetic Algorithms, pp. 51–60 (1989)Google Scholar
  3. 3.
    Conover, W.J.: Practical Nonparametric Statistics. Wiley, New York (1999)Google Scholar
  4. 4.
    Sheskin, D.: Handbook of Parametric and Nonparametric Statistical Procedures, 5th edn. Chapman and Hall/CRC, New York (2011)zbMATHGoogle Scholar
  5. 5.
    Birattari, M., Stutzle, T., Paquete, L., Varrentrapp, K.: A Racing algorithm for configuring metaheuristics. In: Genetic and Evolutionary Computation Conference, pp. 11–18 (2002)Google Scholar
  6. 6.
    Rossi-Doria, O., et al.: A comparison of the performance of different metaheuristics on the timetabling problem. In: Burke, E.K., De Causmaecker, P. (eds.) PATAT 2003. LNCS, vol. 2740, pp. 329–351. Springer, Heidelberg (2003)Google Scholar
  7. 7.
    Paquete, L., Stutzle, T.: A study of stochastic local search algorithms for the biobjective quadratic assignment problem with correlated flow matrices. Eur. J. Oper. Res. 169(3), 943–959 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Balaprakash, P., Birattari, M., Stützle, T., Dorigo, M.: Adaptive sample size and importance sampling in estimation-based local search for the probabilistic traveling salesman problem. Eur. J. Oper. Res. 199(1), 98–110 (2009)CrossRefzbMATHGoogle Scholar
  9. 9.
    Chiarandini, M., Stutzle, T.: Stochastic local search algorithms for graph set T-colouring and frequency assignment. Constraints 12(3), 371–403 (2007)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Di Gaspero, L., Roli, A.: Stochastic local search for large-scale instances of the haplotype inference problem by pure parsimony. J. Algorithms 63(3), 55–69 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Lenne, R., Solnon, C., Stutzle, T., Tannier, E., Birattari, M.: Reactive stochastic local search algorithms for the genomic median problem. In: European Conference on Evolutionary Computation in Combinatorial Optimization, pp. 266–276 (2008)Google Scholar
  12. 12.
    Di Gaspero, L., di Tollo, G., Roli, A., Schaerf, A.: Hybrid local search for constrained financial portfolio selection problems. In: Van Hentenryck, P., Wolsey, L.A. (eds.) CPAIOR 2007. LNCS, vol. 4510, pp. 44–58. Springer, Heidelberg (2007)Google Scholar
  13. 13.
    Blum, C., Socha, K.: Training feed-forward neural networks with ant colony optimization: an application to pattern classification, p. 6Google Scholar
  14. 14.
    Yuan, B., Gallagher, M.: Combining meta-EAs and Racing for difficult EA parameter tuning tasks. In: Lobo, F., Lima, C., Michalewicz, Z. (eds.) Parameter Setting in Evolutionary Algorithms, pp. 121–142. Studies in Computational IntelligenceSpringer, Berlin (2007)CrossRefGoogle Scholar
  15. 15.
    Birattari, M., Balaprakash, P., Dorigo, M.: The ACO/F-Race algorithm for combinatorial optimization under uncertainty. In: Doerner, K., Gendreau, M., Greistorfer, P., Gutjahr, W., Hartl, R., Reimann, M. (eds.) Metaheuristics, Operations Research/Computer Science Interfaces Series, pp. 189–203. Springer, Heidelberg (2007)Google Scholar
  16. 16.
    Yuan, Z., Stützle, T., Birattari, M.: MADS/F-Race: mesh adaptive direct search meets F-Race. In: García-Pedrajas, N., Herrera, F., Fyfe, C., Benítez, J., Ali, M. (eds.) IEA/AIE 2010, Part I. LNCS, vol. 6096, pp. 41–50. Springer, Heidelberg (2010)Google Scholar
  17. 17.
    Yuan, Z., Montes de Oca, M., Birattari, M., Stützle, T.: Continuous optimization algorithms for tuning real and integer parameters of swarm intelligence algorithms. Swarm Intell. 6(1), 49–75 (2012)CrossRefGoogle Scholar
  18. 18.
    Chiarandini, M., Birattari, M., Socha, K., Rossi-Doria, O.: An effective hybrid algorithm for university course timetabling. J. Sched. 9(5), 403–432 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Birattari, M., Yuan, Z., Balaprakash, P., Stützle, T.: F-Race and iterated F-Race: an overview. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 311–336. Springer, Berlin (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Warwick Business SchoolCoventryUK

Personalised recommendations