Quantifying the Similarity of Algorithm Configurations

  • Lin Xu
  • Ashiqur R. KhudaBukhsh
  • Holger H. HoosEmail author
  • Kevin Leyton-BrownEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10079)


A natural way of attacking a new, computationally challenging problem is to find a novel way of combining design elements introduced in existing algorithms. For example, this approach was made systematic in SATenstein [15], a highly parameterized stochastic local search (SLS) framework for SAT that unifies techniques across a wide range of well-known SLS solvers. The focus of such work so far has been on building frameworks and identifying high-performing configurations. Here, we focus on analyzing such frameworks, a problem that currently requires considerable manual effort and domain expertise. We propose a quantitative alternative: a new metric that measures the similarity between a new configuration and previously known algorithm designs. We first introduce concept DAGs, a data structure that preserves the hierarchical structure of configurations induced by conditional parameter dependencies. We then quantify the degree of similarity between two configurations as the transformation cost between the respective concept DAGs. In the context of analyzing SATenstein configurations, we demonstrate that visualizations based on transformation costs can provide useful insights into the similarities and differences between existing SLS-based SAT solvers and novel solver configurations.


SAT Stochastic local search Algorithm configuration similarity 


  1. 1.
    Clarke, E., Kroening, D., Lerda, F.: A tool for checking ANSI-C programs. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 168–176. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-24730-2_15 CrossRefGoogle Scholar
  2. 2.
    Cox, T.F., Cox, M.A.: Multidimensional scaling. CRC Press, Boca Raton (2000)zbMATHGoogle Scholar
  3. 3.
    Eén, N., Biere, A.: Effective preprocessing in SAT through variable and clause elimination. In: Bacchus, F., Walsh, T. (eds.) SAT 2005. LNCS, vol. 3569, pp. 61–75. Springer, Heidelberg (2005). doi: 10.1007/11499107_5 CrossRefGoogle Scholar
  4. 4.
    Fawcett, C., Hoos, H.H.: Analysing differences between algorithm configurations through ablation. J. Heuristics 22, 1–28 (2013)Google Scholar
  5. 5.
    Gent, I.P., Hoos, H.H., Prosser, P., Walsh, T.: Morphing: combining structure and randomness. In: Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI 1999), pp. 654–660 (1999)Google Scholar
  6. 6.
    Gomes, C.P., Selman, B.: Problem structure in the presence of perturbations. In: Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI 1997), pp. 221–226 (1997)Google Scholar
  7. 7.
    Hirsch, E.A.: Random generator hgen2 of satisfiable formulas in 3-CNF (2002). Accessed 18 December 2015
  8. 8.
    Hoos, H.H.: An adaptive noise mechanism for WalkSAT. In: Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI 2002), pp. 655–660 (2002)Google Scholar
  9. 9.
    Hoos, H.H.: Programming by optimization. Commun. ACM 55(2), 70–80 (2012)CrossRefGoogle Scholar
  10. 10.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-25566-3_40 CrossRefGoogle Scholar
  11. 11.
    Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intell. Res. (JAIR) 36(1), 267–306 (2009)zbMATHGoogle Scholar
  12. 12.
    Hutter, F., Tompkins, D.A.D., Hoos, H.H.: Scaling and probabilistic smoothing: efficient dynamic local search for SAT. In: Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, pp. 233–248. Springer, Heidelberg (2002). doi: 10.1007/3-540-46135-3_16 CrossRefGoogle Scholar
  13. 13.
    Hutter, F., Hoos, H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pp. 754–762 (2014)Google Scholar
  14. 14.
    Kadioglu, S., Malitsky, Y., Sellmann, M., Tierney, K.: ISAC - instance specific algorithm configuration. Eur. Conf. Artif. Intell. (ECAI) 2010, 751–756 (2010)Google Scholar
  15. 15.
    KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: SATenstein: automatically building local search SAT solvers from components. In: Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI 2009), pp. 517–524 (2009)Google Scholar
  16. 16.
    KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: Satenstein: automatically building local search sat solvers from components. Artif. Intell. 232, 20–42 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Li, C.M., Huang, W.Q.: Diversification and determinism in local search for satisfiability. In: Bacchus, F., Walsh, T. (eds.) SAT 2005. LNCS, vol. 3569, pp. 158–172. Springer, Heidelberg (2005). doi: 10.1007/11499107_12 CrossRefGoogle Scholar
  18. 18.
    Li, C.M., Wei, W., Zhang, H.: Combining adaptive noise and promising decreasing variables in local search for SAT (2007). Solver description, SAT competition 2007Google Scholar
  19. 19.
    Li, C.M., Wei, W., Zhang, H.: Combining adaptive noise and look-ahead in local search for SAT. In: Marques-Silva, J., Sakallah, K.A. (eds.) SAT 2007. LNCS, vol. 4501, pp. 121–133. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-72788-0_15 CrossRefGoogle Scholar
  20. 20.
    Malitsky, Y., Sabharwal, A., Samulowitz, H., Sellmann, M.: Algorithm portfolios based on cost-sensitive hierarchical clustering. In: Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pp. 608–614. AAAI Press (2013)Google Scholar
  21. 21.
    Nikolić, M., Marić, F., Janičić, P.: Instance-based selection of policies for SAT solvers. In: Kullmann, O. (ed.) SAT 2009. LNCS, vol. 5584, pp. 326–340. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02777-2_31 CrossRefGoogle Scholar
  22. 22.
    Pham, D.N., Anbulagan, A.: Resolution enhanced SLS solver: R+AdaptNovelty+ (2007). Solver description, SAT competition 2007Google Scholar
  23. 23.
    Pham, D.N., Thornton, J., Gretton, C., Sattar, A.: Combining adaptive and dynamic local search for satisfiability. J. Satisf. Boolean Model. Comput. (JSAT) 4, 149–172 (2008)zbMATHGoogle Scholar
  24. 24.
    Prestwich, S.: Random walk with continuously smoothed variable weights. In: Bacchus, F., Walsh, T. (eds.) SAT 2005. LNCS, vol. 3569, pp. 203–215. Springer, Heidelberg (2005). doi: 10.1007/11499107_15 CrossRefGoogle Scholar
  25. 25.
    Simon, L.: SAT competition random 3CNF generator (2002). Accessed 18 December 2015
  26. 26.
    Sinz, C.: Visualizing the internal structure of sat instances (preliminary report). In: SAT. Citeseer (2004)Google Scholar
  27. 27.
    Thornton, J., Pham, D.N., Bain, S., Ferreira, V.: Additive versus multiplicative clause weighting for SAT. In: Proceedings of the Nineteenth National Conference on Artificial Intelligence (AAAI 2004), pp. 191–196 (2004)Google Scholar
  28. 28.
    Tompkins, D.A.D., Balint, A., Hoos, H.H.: Captain Jack: new variable selection heuristics in local search for SAT. In: Sakallah, K.A., Simon, L. (eds.) SAT 2011. LNCS, vol. 6695, pp. 302–316. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21581-0_24 CrossRefGoogle Scholar
  29. 29.
    Uchida, T., Watanabe, O.: Hard SAT instance generation based on the factorization problem (1999).
  30. 30.
    Wei, W., Li, C.M., Zhang, H.: A switching criterion for intensification, and diversification in local search for sat. J. Satisf. Boolean Model. Comput. 4, 219–237 (2008)zbMATHGoogle Scholar
  31. 31.
    Xue, Y., Wang, C., Ghenniwa, H., Shen, W.: A tree similarity measuring method and its application to ontology. J. Univ. Comput. Sci. 15(9), 1766–1781 (2001)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.University of British ColumbiaVancouverCanada
  2. 2.Carnegie Mellon UniversityPittsburghUSA

Personalised recommendations