Embedding Decision Trees and Random Forests in Constraint Programming

  • Alessio Bonfietti
  • Michele Lombardi
  • Michela Milano
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9075)

Abstract

In past papers, we have introduced Empirical Model Learning (EML) as a method to enable Combinatorial Optimization on real world systems that are impervious to classical modeling approaches. The core idea in EML consists in embedding a Machine Learning model in a traditional combinatorial model. So far, the method has been demonstrated by using Neural Networks and Constraint Programming (CP). In this paper we add one more technique to the EML arsenal, by devising methods to embed Decision Trees (DTs) in CP. In particular, we propose three approaches: 1) a simple encoding based on meta-constraints; 2) a method using attribute discretization and a global table constraint; 3) an approach based on converting a DT into a Multi-valued Decision Diagram, which is then fed to an mdd constraint. We finally show how to embed in CP a Random Forest, a powerful type of ensemble classifier based on DTs. The proposed methods are compared in an experimental evaluation, highlighting their strengths and their weaknesses.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bartolini, A., Cacciari, M., Tilli, A., Benini, L.: Thermal and energy management of high-performance multicores: Distributed and self-calibrating model-predictive controller. IEEE Trans. Parallel Distrib. Syst. 24(1), 170–183 (2013)CrossRefGoogle Scholar
  2. 2.
    Bartolini, A., Lombardi, M., Milano, M., Benini, L.: Neuron constraints to model complex real-world problems. In: Proc. of CP, pp. 115–129 (2011)Google Scholar
  3. 3.
    Battiti, R., Brunato, M.: The LION way: Machine Learning plus Intelligent Optimization. University of Trento, LIONlab (2014)Google Scholar
  4. 4.
    Beldiceanu, N., Simonis, H.: A model seeker: extracting global constraint models from positive examples. In: Proc. of CP, pp. 141–157 (2012)Google Scholar
  5. 5.
    Bennett, K.P., Parrado-Hernández, E.: The interplay of optimization and machine learning research. Journal of Machine Learning Research 7, 1265–1281 (2006)MATHGoogle Scholar
  6. 6.
    Benoist, T., Estellon, B., Gardi, F., Megel, R., Nouioua, K.: Localsolver 1.x: a black-box local-search solver for 0–1 programming. 4OR 9(3), 299–316 (2011)CrossRefMATHMathSciNetGoogle Scholar
  7. 7.
    Bessière, C., Coletta, R., Freuder, E.C., O’Sullivan, B.: Leveraging the learning power of examples in automated constraint acquisition. In: Proc. of CP, pp. 123–137 (2004)Google Scholar
  8. 8.
    Bessière, C., Coletta, R., Hebrard, E., Katsirelos, G., Lazaar, N., Narodytska, N., Quimper, C., Walsh, T.: Constraint acquisition via partial queries. In: Proc. of IJCAI (2013)Google Scholar
  9. 9.
    Bessière, C., Coletta, R., O’Sullivan, B., Paulin, M.: Query-driven constraint acquisition. In: Proc. of IJCAI, pp. 50–55 (2007)Google Scholar
  10. 10.
    Breiman, L.: Random forests. Machine learning 45(1), 5–32 (2001)CrossRefMATHGoogle Scholar
  11. 11.
    Cheng, K.C.K., Yap, R.H.C.: An mdd-based generalized arc consistency algorithm for positive and negative table constraints and some global constraints. Constraints 15(2), 265–304 (2010)CrossRefMATHMathSciNetGoogle Scholar
  12. 12.
    Gardi, F., Benoist, T., Darlay, J., Estellon, B., Megel, R.: Mathematical Programming Solver Based on Local Search. John Wiley & Sons (2014)Google Scholar
  13. 13.
    Gent, I.P., Jefferson, C., Miguel, I., Nightingale, P.: Data structures for generalised arc consistency for extensional constraints. In: Proc. of AAAI, pp. 191–197 (2007)Google Scholar
  14. 14.
    Glover, F., Kelly, J.P., Laguna, M.: New Advances for Wedding optimization and simulation. In: Proc. of WSC, pp. 255–260 (1999)Google Scholar
  15. 15.
    Gopalakrishnan, K., Asce, A.M.: Neural Network - Swarm Intelligence Hybrid Nonlinear Optimization Algorithm for Pavement Moduli Back-Calculation. Journal of Transportation Engineering 136(6), 528–536 (2009)CrossRefGoogle Scholar
  16. 16.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: an update. ACM SIGKDD explorations newsletter 11(1), 10–18 (2009)CrossRefGoogle Scholar
  17. 17.
    Hernando, Leticia, Mendiburu, Alexander, Lozano, Jose A.: Generating Customized Landscapes in Permutation-Based Combinatorial Optimization Problems. In: Nicosia, Giuseppe, Pardalos, Panos (eds.) LION 7. LNCS, vol. 7997, pp. 299–303. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  18. 18.
    Ho, T.K.: Random decision forests. In: Proc. of ICDAR, p. 278 (1995)Google Scholar
  19. 19.
    Howard, J., Dighe, S., et al.: A 48-Core IA-32 message-passing processor with DVFS in 45nm CMOS. In: Proc. of ISSCC, pp. 108–109, February 2010Google Scholar
  20. 20.
    Hutter, F., Xu, L., Hoos, H.H., Leyton-Brown, K.: Algorithm runtime prediction: Methods & evaluation. Artif. Intell. 206, 79–111 (2014)CrossRefMathSciNetGoogle Scholar
  21. 21.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. Journal of Global optimization 13(4), 455–492 (1998)CrossRefMATHMathSciNetGoogle Scholar
  22. 22.
    Kadioglu, S., Malitsky, Y., Sellmann, M., Tierney, K.: ISAC - instance-specific algorithm configuration. In: Proc. of ECAI, pp. 751–756 (2010)Google Scholar
  23. 23.
    Katsirelos, G., Walsh, T.: A compression algorithm for large arity extensional constraints. In: Proc. of CP, pp. 379–393 (2007)Google Scholar
  24. 24.
    Kotthoff, L., Gent, I.P., Miguel, I.: An evaluation of machine learning in algorithm selection for search problems. AI Commun. 25(3), 257–270 (2012)MathSciNetGoogle Scholar
  25. 25.
    Perron, L.: Operations Research and Constraint Programming at Google. In: Proc. of CP, p. 2 (2011)Google Scholar
  26. 26.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann (1993)Google Scholar
  27. 27.
    Shaw, P.: Using constraint programming and local search methods to solve vehicle routing problems. In: Proc. of CP, pp. 417–431 (1998)Google Scholar
  28. 28.
    Sra, S., Nowozin, S., Wright, S.J.: Optimization for machine learning. MIT Press(2012)Google Scholar
  29. 29.
    Zaabab, A.H., Zhang, Q., Nakhla, M.: A neural network modeling approach to circuit optimization and statistical design. IEEE Transactions on Microwave Theory and Techniques 43(6), 1349–1358 (1995)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Alessio Bonfietti
    • 1
  • Michele Lombardi
    • 1
  • Michela Milano
    • 1
  1. 1.DISIUniversity of BolognaBolognaItaly

Personalised recommendations