Constraints

, Volume 21, Issue 4, pp 435–462 | Cite as

A lagrangian propagator for artificial neural networks in constraint programming

Article

Abstract

This paper discusses a new method to perform propagation over a (two-layer, feed-forward) Neural Network embedded in a Constraint Programming model. The method is meant to be employed in Empirical Model Learning, a technique designed to enable optimal decision making over systems that cannot be modeled via conventional declarative means. The key step in Empirical Model Learning is to embed a Machine Learning model into a combinatorial model. It has been showed that Neural Networks can be embedded in a Constraint Programming model by simply encoding each neuron as a global constraint, which is then propagated individually. Unfortunately, this decomposition approach may lead to weak bounds. To overcome such limitation, we propose a new network-level propagator based on a non-linear Lagrangian relaxation that is solved with a subgradient algorithm. The method proved capable of dramatically reducing the search tree size on a thermal-aware dispatching problem on multicore CPUs. The overhead for optimizing the Lagrangian multipliers is kept within a reasonable level via a few simple techniques. This paper is an extended version of [27], featuring an improved structure, a new filtering technique for the network inputs, a set of overhead reduction techniques, and a thorough experimentation.

Keywords

Constraint programming Lagrangian relaxation Neural networks 

References

  1. 1.
    Audet, C. (2014). A survey on direct search methods for blackbox optimization and their applications. In Mathematics Without Boundaries (pp. 31–56): Springer.Google Scholar
  2. 2.
    Bartolini, A., Lombardi, M., Milano, M., & Benini, L. (2011). Neuron Constraints to Model Complex Real-World Problems. In Proc. of CP (pp. 115–129).Google Scholar
  3. 3.
    Bartolini, A., Lombardi, M., Milano, M., & Benini, L. (2012). Optimization and Controlled Systems: A Case Study on Thermal Aware Workload Dispatching. Proc. of AAAI.Google Scholar
  4. 4.
    Basheer, I.A., & Hajmeer, M. (2000). Artificial neural networks: fundamentals, computing, design, and application. Journal of Microbiological Methods, 43(1), 3–31.CrossRefGoogle Scholar
  5. 5.
    Belew, R.K., McInerney, J., & Schraudolph, N.N. (1991). Evolving networks: Using the genetic algorithm with connectionist learning. Proc. of Artificial Life, 511–547.Google Scholar
  6. 6.
    Belotti, P., Lee, J., Liberti, L., Margot, F., & Wächter, A. (2009). Branching and bounds tightening techniques for non-convex MINLP. Optimization Methods and Software, 24(4-5), 597–634.MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Bergman, D., Cirė, A. A., & van Hoeve, W.-J. (2015). Lagrangian bounds from decision diagrams. Constraints, 20(3), 346–361.MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Bonfietti, A., & Lombardi, M. (2012). The weighted average constraint. In Proc. of CP (pp. 191–206): Springer.Google Scholar
  9. 9.
    Bonfietti, A., Lombardi, M., & Milano, M. (2015). Embedding decision trees and random forests in constraint programming. In Proc. of CPAIOR (pp. 74–90).Google Scholar
  10. 10.
    Cambazard, H., & Fages, J.-G. (2015). New filtering for atmostnvalue and its weighted variant: A lagrangian approach. Constraints, 20(3), 362–380.MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Chow, T.T., Zhang, G.Q., Lin, Z., & Song, C.L. (2002). Global optimization of absorption chiller system by genetic algorithm and neural network. Energy and Buildings, 34(1), 103–109.CrossRefGoogle Scholar
  12. 12.
    Conn, A.R., Scheinberg, K., & Vicente, L.N. (2009). Introduction To Derivative-free Optimization, volume 8. Siam.Google Scholar
  13. 13.
    d’Antonio, G., & Frangioni, A. (2009). Convergence analysis of deflected conditional approximate subgradient methods. SIAM Journal on Optimization, 20(1), 357–386.MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Focacci, F., Lodi, A., & Milano, M. (1999). Cost-based domain filtering.Google Scholar
  15. 15.
    Ge, S.S., Hang, C.C., Lee, T.H., & Zhang, T. (2010). Stable adaptive neural network control. Springer Publishing Company, Incorporated.Google Scholar
  16. 16.
    Gent, I.P., Kotthoff, L., Miguel, I., & Nightingale, P. (2010). Machine learning for constraint solver design – A case study for the alldifferent constraint. CoRR, abs/1008.4326.Google Scholar
  17. 17.
    Glover, F., Kelly, J.P., & Laguna, M. (1999). New Advances for Wedding optimization and simulation. In Proc. of WSC. IEEE (pp. 255–260).Google Scholar
  18. 18.
    Gopalakrishnan, K., & Asce, A.M. (2009). Neural Network Swarm Intelligence Hybrid Nonlinear Optimization Algorithm for Pavement Moduli Back-Calculation. Journal of Transportation Engineering, 136(6), 528–536.CrossRefGoogle Scholar
  19. 19.
    Gualandi, S., & Malucelli, F. (2012). Resource constrained shortest paths with a super additive objective function. In Proc. of CP (pp. 299–315): Springer.Google Scholar
  20. 20.
    Howard, J., Dighe, S., Vangal, S.R., Ruhl, G., Borkar, N., Jain, S., Erraguntla, V., Konow, M., Riepen, M., Gries, M., & et al. (2011). A 48-core ia-32 processor in 45 nm cmos using on-die message-passing and dvfs for performance and power scaling. IEEE Journal of Solid-State Circuits, 46(1), 173–183.CrossRefGoogle Scholar
  21. 21.
    Huang, W., Ghosh, S., & Velusamy, S. (2006). HotSpot: A compact thermal modeling methodology for early-stage VLSI design. IEEE Transactions on VLSI, 14 (5), 501–513.CrossRefGoogle Scholar
  22. 22.
    Hutter, F., Hoos, H. H., Leyton-Brown, K., & Stu̇tzle, T. (2009). Paramils: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36, 267–306.MATHGoogle Scholar
  23. 23.
    Jayaseelan, R., & Mitra, T. (2009). A hybrid local-global approach for multi-core thermal management. In Proc. of ICCAD (pp. 314–320): ACM Press.Google Scholar
  24. 24.
    Kiranyaz, S., Ince, T., Yildirim, A., & Gabbouj, M. (2009). Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Networks, 22(10), 1448–1462.CrossRefGoogle Scholar
  25. 25.
    Lemaréchal, C. (2001). Lagrangian relaxation. In Computational Combinatorial Optimization (pp. 112–156): Springer.Google Scholar
  26. 26.
    Ljung, L. (1999). System identification. Wiley Online Library.Google Scholar
  27. 27.
    Lombardi, M., & Gualandi, S. (2013). A new propagator for two-layer neural networks in empirical model learning. In Proc. of CP (pp. 448–463).Google Scholar
  28. 28.
    Montana, D.J., & Davis, L. (1989). Training feedforward neural networks using genetic algorithms. In Proc. of IJCAI (pp. 762–767).Google Scholar
  29. 29.
    Moore, J., Chase, J.S., & Ranganathan, P. (2006). Weatherman: Automated, Online and Predictive Thermal Mapping and Management for Data Centers. In Proc. of ICAC. IEEE (pp. 155–164).Google Scholar
  30. 30.
    Moré, J.J. (1978). The Levenberg-Marquardt algorithm: implementation and theory. In Numerical analysis (pp. 105–116): Springer.Google Scholar
  31. 31.
    Queipo, N.V., Haftka, R.T., Shyy, W., Goel, T., Vaidyanathan, R., & Tucker, P.K. (2005). Surrogate-based analysis and optimization. Progress In Aerospace Sciences, 41(1), 1–28.CrossRefGoogle Scholar
  32. 32.
    Sellmann, M. (2004). Theoretical foundations of cp-based lagrangian relaxation. In Proc. of CP (pp. 634–647): Springer.Google Scholar
  33. 33.
    Sellmann, M., & Fahle, T. (2003). Constraint programming based lagrangian relaxation for the automatic recording problem. Annals of Operations Research, 118 (1–4), 17–33.MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Slusky, M.R., & van Hoeve, W.J. (2013). A lagrangian relaxation for golomb rulers. In Proc. of CPAIOR (pp. 251–267): Springer.Google Scholar
  35. 35.
    Van Cauwelaert, S., Lombardi, M., & Schaus, P. (2015). Understanding the potential of propagators. In Proc. of CPAIOR (pp. 427–436).Google Scholar
  36. 36.
    Zhang, G., Patuwo, B.E., & Hu, M.Y. (1998). Forecasting with artificial neural networks: The state of the art. International Journal of Forecasting, 14(1), 35–62.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.University of BolognaBolognaItaly
  2. 2.AntOptima SALuganoSwitzerland

Personalised recommendations