Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4204)


Machine learning can be used to build models that predict the run-time of search algorithms for hard combinatorial problems. Such empirical hardness models have previously been studied for complete, deterministic search algorithms. In this work, we demonstrate that such models can also make surprisingly accurate predictions of the run-time distributions of incomplete and randomized search methods, such as stochastic local search algorithms. We also show for the first time how information about an algorithm’s parameter settings can be incorporated into a model, and how such models can be used to automatically adjust the algorithm’s parameters on a per-instance basis in order to optimize its performance. Empirical results for Novelty +  and SAPS on structured and unstructured SAT instances show very good predictive performance and significant speedups of our automatically determined parameter settings when compared to the default and best fixed distribution-specific parameter settings.


Root Mean Square Error Local Search Gaussian Process Regression Hard Instance Stochastic Local Search 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adenso-Díaz, B., Laguna, M.: Fine-tuning of algorithms using fractional experimental design and local search. Operations Research 54(1) (to appear, 2006)Google Scholar
  2. 2.
    Battiti, R., Brunato, M.: Reactive search: machine learning for memory-based heuristics. Technical Report DIT-05-058, Università Degli Studi Di Trento, Dept. of information and communication technology, Trento, Italy (September 2005)Google Scholar
  3. 3.
    Birattari, M., Stützle, T., Paquete, L., Varrentrapp, K.: A racing algorithm for configuring metaheuristics. In: Proc. of GECCO 2002, pp. 11–18 (2002)Google Scholar
  4. 4.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)Google Scholar
  5. 5.
    Brglez, F., Li, X.Y., Stallmann, M.F.: On SAT instance classes and a method for reliable performance experiments with SAT solvers. Annals of Mathematics and Artificial Intelligence, 1–34 (2004)Google Scholar
  6. 6.
    Carchrae, T., Beck, J.C.: Applying machine learning to low-knowledge control of optimization algorithms. Computational Intelligence 21(4), 372–387 (2005)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Cohn, D.A., Ghahramani, Z., Jordan, M.I.: Active learning with statistical models. JAIR 4, 129–145 (1996)zbMATHGoogle Scholar
  8. 8.
    Gebruers, C., Hnich, B., Bridge, D.G., Freuder, E.C.: Using CBR to select solution strategies in constraint programming. In: Muñoz-Ávila, H., Ricci, F. (eds.) ICCBR 2005. LNCS, vol. 3620, pp. 222–236. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Gent, I.P., Hoos, H.H., Prosser, P., Walsh, T.: Morphing: Combining structure and randomness. In: Proc. of AAAI 1999, Orlando, Florida, pp. 654–660 (1999)Google Scholar
  10. 10.
    Gomes, C.P., Selman, B.: Problem structure in the presence of perturbations. In: Proc. of AAAI 1997 (1997)Google Scholar
  11. 11.
    Gomes, C.P., Selman, B., Crato, N., Kautz, H.: Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. J. of Automated Reasoning 24(1) (2000)Google Scholar
  12. 12.
    Hoos, H.H.: On the run-time behaviour of stochastic local search algorithms for SAT. In: Proc. of AAAI 1999, pp. 661–666 (1999)Google Scholar
  13. 13.
    Hoos, H.H.: An adaptive noise mechanism for WalkSAT. In: Proc. of AAAI 2002, pp. 655–660 (2002)Google Scholar
  14. 14.
    Hoos, H.H., Stützle, T.: Stochastic Local Search - Foundations & Applications. Morgan Kaufmann, San Francisco (2004)Google Scholar
  15. 15.
    Horvitz, E., Ruan, Y., Gomes, C.P., Kautz, H., Selman, B., Chickering, D.M.: A Bayesian approach to tackling hard computational problems. In: Proc.  of UAI 2001 (2001)Google Scholar
  16. 16.
    Hutter, F., Hamadi, Y.: Parameter adjustment based on performance prediction: Towards an instance-aware problem solver. Technical Report MSR-TR-2005-125, Microsoft Research, Cambridge, UK (December 2005)Google Scholar
  17. 17.
    Hutter, F., Tompkins, D.A.D., Hoos, H.H.: Scaling and probabilistic smoothing: Efficient dynamic local search for SAT. In: Van Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, pp. 233–248. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  18. 18.
    Lagoudakis, M.G., Littman, M.L.: Learning to select branching rules in the DPLL procedure for satisfiability. In: Electronic Notes in Discrete Mathematics, ENDM (2001)Google Scholar
  19. 19.
    Leyton-Brown, K., Nudelman, E., Shoham, Y.: Learning the empirical hardness of optimization problems: The case of combinatorial auctions. In: Van Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, p. 556. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  20. 20.
    McAllester, D., Selman, B., Kautz, H.: Evidence for invariants in local search. In: Proc. of AAAI 1997, pp. 321–326 (1997)Google Scholar
  21. 21.
    Nudelman, E., Leyton-Brown, K., Hoos, H.H., Devkar, A., Shoham, Y.: Understanding random SAT: Beyond the clauses-to-variables ratio. In: Wallace, M. (ed.) CP 2004. LNCS, vol. 3258, pp. 438–452. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  22. 22.
    Patterson, D.J., Kautz, H.: Auto-WalkSAT: a self-tuning implementation of WalkSAT. In: Electronic Notes in Discrete Mathematics (ENDM), vol. 9 (2001)Google Scholar
  23. 23.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  24. 24.
    Srivastava, B., Mediratta, A.: Domain-dependent parameter selection of search-based algorithms compatible with user performance criteria. In: Proc. of AAAI 2005 (2005)Google Scholar
  25. 25.
    Thornton, J.R.: Clause weighting local search for SAT. J. of Automated Reasoning (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  1. 1.University of British ColumbiaVancouverCanada
  2. 2.Microsoft ResearchCambridgeUK

Personalised recommendations