Learning Heuristic Policies – A Reinforcement Learning Problem

  • Thomas Philip Runarsson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6683)


How learning heuristic policies may be formulated as a reinforcement learning problem is discussed. Reinforcement learning algorithms are commonly centred around estimating value functions. Here a value function represents the average performance of the learned heuristic algorithm over a problem domain. Heuristics correspond to actions and states to solution instances. The problem of bin packing is used to illustrate the key concepts. Experimental studies show that the reinforcement learning approach is compatible with the current techniques used for learning heuristics. The framework opens up further possibilities for learning heuristics by exploring the numerous techniques available in the reinforcement learning literature.


Problem Instance Problem Domain Reinforcement Learning Algorithm Construction Heuristic Solution Instance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bai, R., Burke, E.K., Gendreau, M., Kendall, G., McCollum, B.: Memory length in hyper-heuristics: An empirical study. In: IEEE Symposium on Computational Intelligence in Scheduling, SCIS 2007, pp. 173–178. IEEE, Los Alamitos (2007)CrossRefGoogle Scholar
  2. 2.
    Burke, E.K., Hyde, M.R., Kendall, G., Ochoa, G., Ozcan, E., Woodward, J.R.: Exploring hyper-heuristic methodologies with genetic programming. Computational Intelligence, 177–201 (2009)Google Scholar
  3. 3.
    Burke, E.K., Hyde, M.R., Kendall, G., Woodward, J.: A genetic programming hyperheuristic approach for evolving two dimensional strip packing heuristics. IEEE Transactions on Evolutionary Computation (2010) (to appear)Google Scholar
  4. 4.
    Burke, E.K., Kendall, G.: Search methodologies: introductory tutorials in optimization and decision support techniques. Springer, Heidelberg (2005)CrossRefzbMATHGoogle Scholar
  5. 5.
    Burker, E.K., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., Woodward, J.R.: A classification of hyper-heuristic approaches. In: Handbook of Metaheuristics, pp. 449–468 (2010)Google Scholar
  6. 6.
    Crescenzi, P., Kann, V.: A compendium of NP optimization problems. Technical report, (accessed September 2010)
  7. 7.
    Dorigo, M., Gambardella, L.: A study of some properties of Ant-Q. In: Ebeling, W., Rechenberg, I., Voigt, H.-M., Schwefel, H.-P. (eds.) PPSN 1996. LNCS, vol. 1141, pp. 656–665. Springer, Heidelberg (1996)CrossRefGoogle Scholar
  8. 8.
    Dorigo, M., Gambardella, L.M.: Ant colony system: A cooperative learning approach to the traveling salesman problem. IEEE Transactions on Evolutionary Computation 1(1), 53–66 (2002)CrossRefGoogle Scholar
  9. 9.
    Floyd, S., Karp, R.M.: FFD bin packing for item sizes with uniform distributions on [0, 1/2]. Algorithmica 6(1), 222–240 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Meignan, D., Koukam, A., Créput, J.C.: Coalition-based metaheuristic: a self-adaptive metaheuristic using reinforcement learning and mimetism. Journal of Heuristics, 1–21Google Scholar
  11. 11.
    Nareyek, A.: Choosing search heuristics by non-stationary reinforcement learning. Applied Optimization 86, 523–544 (2003)CrossRefGoogle Scholar
  12. 12.
    Poli, R., Woodward, J., Burke, E.K.: A histogram-matching approach to the evolution of bin-packing strategies. In: IEEE Congress on Evolutionary Computation, CEC 2007, pp. 3500–3507 (September 2007)Google Scholar
  13. 13.
    Powell, W.B.: Approximate Dynamic Programming: Solving the curses of dimensionality. Wiley-Interscience, Hoboken (2007)CrossRefzbMATHGoogle Scholar
  14. 14.
    Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. The MIT Press, Cambridge (1998)Google Scholar
  15. 15.
    Zhang, W., Dietterich, T.G.: A Reinforcement Learning Approach to Job-shop Scheduling. In: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1114–1120. Morgan Kaufmann, San Francisco (1995)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Thomas Philip Runarsson
    • 1
  1. 1.School of Engineering and Natural SciencesUniversity of IcelandIceland

Personalised recommendations