Approximate Dynamic Programming and Reinforcement Learning

  • Lucian Buşoniu
  • Bart De Schutter
  • Robert Babuška
Part of the Studies in Computational Intelligence book series (SCI, volume 281)

Abstract

Dynamic programming (DP) and reinforcement learning (RL) can be used to address problems from a variety of fields, including automatic control, artificial intelligence, operations research, and economy. Many problems in these fields are described by continuous variables, whereas DP and RL can find exact solutions only in the discrete case. Therefore, approximation is essential in practical DP and RL. This chapter provides an in-depth review of the literature on approximate DP and RL in large or continuous-space, infinite-horizon problems. Value iteration, policy iteration, and policy search approaches are presented in turn. Model-based (DP) as well as online and batch model-free (RL) algorithms are discussed. We review theoretical guarantees on the approximate solutions produced by these algorithms. Numerical examples illustrate the behavior of several representative algorithms in practice. Techniques to automatically derive value function approximators are discussed, and a comparison between value iteration, policy iteration, and policy search is provided. The chapter closes with a discussion of open issues and promising research directions in approximate DP and RL.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Baddeley, B.: Reinforcement learning in continuous time and space: Interference and not ill conditioning is the main problem when using distributed function approximators. IEEE Transactions on Systems, Man, and Cybernetics—Part B: Cybernetics 38(4), 950–956 (2008)CrossRefGoogle Scholar
  2. 2.
    Barash, D.: A genetic search in policy space for solving Markov decision processes. In: AAAI Spring Symposium on Search Techniques for Problem Solving under Uncertainty and Incomplete Information. Palo Alto, US (1999)Google Scholar
  3. 3.
    Barto, A.G., Sutton, R.S., Anderson, C.W.: Neuronlike adaptive elements than can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics 13(5), 833–846 (1983)Google Scholar
  4. 4.
    Baxter, J., Bartlett, P.L.: Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research 15, 319–350 (2001)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Berenji, H.R., Khedkar, P.: Learning and tuning fuzzy logic controllers through reinforcements. IEEE Transactions on Neural Networks 3(5), 724–740 (1992)CrossRefGoogle Scholar
  6. 6.
    Berenji, H.R., Vengerov, D.: A convergent actor-critic-based FRL algorithm with application to power management of wireless transmitters. IEEE Transactions on Fuzzy Systems 11(4), 478–485 (2003)CrossRefGoogle Scholar
  7. 7.
    Bertsekas, D.P.: Adaptive aggregation methods for infinite horizon dynamic programming. IEEE Transactions on Automatic Control 34(6), 589–598 (1989)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Bertsekas, D.P.: Dynamic programming and suboptimal control: A survey from ADP to MPC. European Journal of Control 11(4-5) (2005); Special issue for the CDC-ECC-05 in Seville, SpainGoogle Scholar
  9. 9.
    Bertsekas, D.P.: Dynamic Programming and Optimal Control, 3rd edn., vol. 2. Athena Scientific, Belmont (2007)Google Scholar
  10. 10.
    Bertsekas, D.P., Shreve, S.E.: Stochastic Optimal Control: The Discrete Time Case. Academic Press, London (1978)MATHGoogle Scholar
  11. 11.
    Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-Dynamic Programming. Athena Scientific, Belmont (1996)MATHGoogle Scholar
  12. 12.
    Borkar, V.: An actor–critic algorithm for constrained Markov decision processes. Systems & Control Letters 54, 207–213 (2005)MATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Buşoniu, L., Babuška, R., De Schutter, B.: A comprehensive survey of multi-agent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics 38(2), 156–172 (2008)CrossRefGoogle Scholar
  14. 14.
    Buşoniu, L., Ernst, D., De Schutter, B., Babuška, R.: Consistency of fuzzy model-based reinforcement learning. In: Proceedings 2008 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2008), Hong Kong, pp. 518–524 (2008)Google Scholar
  15. 15.
    Buşoniu, L., Ernst, D., De Schutter, B., Babuška, R.: Fuzzy partition optimization for approximate fuzzy Q-iteration. In: Proceedings 17th IFAC World Congress (IFAC 2008), Seoul, Korea, pp. 5629–5634 (2008)Google Scholar
  16. 16.
    Buşoniu, L., Ernst, D., De Schutter, B., Babuška, R.: Policy search with cross-entropy optimization of basis functions. In: Proceedings 2009 IEEE International Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL 2009), Nashville, US, pp. 153–160 (2009)Google Scholar
  17. 17.
    Chang, H.S., Fu, M.C., Hu, J., Marcus, S.I.: Simulation-Based Algorithms for Markov Decision Processes. Springer, Heidelberg (2007)MATHGoogle Scholar
  18. 18.
    Chin, H.H., Jafari, A.A.: Genetic algorithm methods for solving the best stationary policy of finite Markov decision processes. In: Proceedings 30th Southeastern Symposium on System Theory, Morgantown, US, pp. 538–543 (1998)Google Scholar
  19. 19.
    Chow, C.S., Tsitsiklis, J.N.: An optimal one-way multigrid algorithm for discrete-time stochastic control. IEEE Transactions on Automatic Control 36(8), 898–914 (1991)MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Coulom, R.: Feedforward neural networks in reinforcement learning applied to high-dimensional motor control. In: Cesa-Bianchi, N., Numao, M., Reischuk, R. (eds.) ALT 2002. LNCS (LNAI), vol. 2533, pp. 403–413. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  21. 21.
    Ernst, D., Geurts, P., Wehenkel, L.: Tree-based batch mode reinforcement learning. Journal of Machine Learning Research 6, 503–556 (2005)MathSciNetGoogle Scholar
  22. 22.
    Ernst, D., Glavic, M., Capitanescu, F., Wehenkel, L.: Reinforcement learning versus model predictive control: a comparison on a power system problem. IEEE Transactions on Systems, Man, and Cybernetics—Part B: Cybernetics 39(2), 517–529 (2009)CrossRefGoogle Scholar
  23. 23.
    Glorennec, P.Y.: Reinforcement learning: An overview. In: Proceedings European Symposium on Intelligent Techniques (ESIT 2000), Aachen, Germany, pp. 17–35 (2000)Google Scholar
  24. 24.
    Gomez, F.J., Schmidhuber, J., Miikkulainen, R.: Efficient non-linear control through neuroevolution. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS (LNAI), vol. 4212, pp. 654–662. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  25. 25.
    Gonzalez, R.L., Rofman, E.: On deterministic control problems: An approximation procedure for the optimal cost I. The stationary problem. SIAM Journal on Control and Optimization 23(2), 242–266 (1985)MATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Gordon, G.: Stable function approximation in dynamic programming. In: Proceedings 12th International Conference on Machine Learning (ICML 1995), Tahoe City, US, pp. 261–268 (1995)Google Scholar
  27. 27.
    Grüne, L.: Error estimation and adaptive discretization for the discrete stochastic Hamilton-Jacobi-Bellman equation. Numerical Mathematics 99, 85–112 (2004)MATHCrossRefGoogle Scholar
  28. 28.
    Horiuchi, T., Fujino, A., Katai, O., Sawaragi, T.: Fuzzy interpolation-based Q-learning with continuous states and actions. In: Proceedings 5th IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 1996), New Orleans, US, pp. 594–600 (1996)Google Scholar
  29. 29.
    Jaakkola, T., Jordan, M.I., Singh, S.P.: On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation 6(6), 1185–1201 (1994)MATHCrossRefGoogle Scholar
  30. 30.
    Jouffe, L.: Fuzzy inference system learning by reinforcement methods. IEEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews 28(3), 338–355 (1998)CrossRefGoogle Scholar
  31. 31.
    Jung, T., Polani, D.: Least squares SVM for least squares TD learning. In: Proceedings of 17th European Conference on Artificial Intelligence (ECAI 2006), Riva del Garda, Italy, pp. 499–503 (2006)Google Scholar
  32. 32.
    Jung, T., Polani, D.: Kernelizing LSPE(λ). In: Proceedings 2007 IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2007), Honolulu, US, pp. 338–345 (2007)Google Scholar
  33. 33.
    Jung, T., Uthmann, T.: Experiments in value function approximation with sparse support vector regression. In: Proceedings 15th European Conference on Machine Learning (ECML 2004), Pisa, Italy, pp. 180–191 (2004)Google Scholar
  34. 34.
    Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101, 99–134 (1998)MATHCrossRefMathSciNetGoogle Scholar
  35. 35.
    Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)Google Scholar
  36. 36.
    Konda, V.: Actor–critic algorithms. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, US (2002)Google Scholar
  37. 37.
    Konda, V.R., Tsitsiklis, J.N.: Actor–critic algorithms. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in Neural Information Processing Systems, vol. 12, pp. 1008–1014. MIT Press, Cambridge (2000)Google Scholar
  38. 38.
    Konda, V.R., Tsitsiklis, J.N.: On actor–critic algorithms. SIAM Journal on Control and Optimization 42(4), 1143–1166 (2003)MATHCrossRefMathSciNetGoogle Scholar
  39. 39.
    Lagoudakis, M., Parr, R., Littman, M.: Least-squares methods in reinforcement learning for control. In: Vlahavas, I.P., Spyropoulos, C.D. (eds.) SETN 2002. LNCS (LNAI), vol. 2308, pp. 249–260. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  40. 40.
    Lagoudakis, M.G., Parr, R.: Least-squares policy iteration. Journal of Machine Learning Research 4, 1107–1149 (2003)CrossRefMathSciNetGoogle Scholar
  41. 41.
    Lagoudakis, M.G., Parr, R.: Reinforcement learning as classification: Leveraging modern classifiers. In: Proceedings 20th International Conference on Machine Learning (ICML 2003), Washington, US, pp. 424–431 (2003)Google Scholar
  42. 42.
    Lewis, R.M., Torczon, V.: Pattern search algorithms for bound constrained minimization. SIAM Journal on Optimization 9(4), 1082–1099 (1999)MATHCrossRefMathSciNetGoogle Scholar
  43. 43.
    Lin, L.J.: Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning 8(3/4), 293–321 (1992); Special Issue on Reinforcement LearningCrossRefGoogle Scholar
  44. 44.
    Liu, D., Javaherian, H., Kovalenko, O., Huang, T.: Adaptive critic learning techniques for engine torque and air-fuel ratio control. IEEE Transactions on Systems, Man, and Cybernetics—Part B: Cybernetics 38(4), 988–993 (2008)CrossRefGoogle Scholar
  45. 45.
    Madani, O.: On policy iteration as a newton s method and polynomial policy iteration algorithms. In: Proceedings 18th National Conference on Artificial Intelligence and 14th Conference on Innovative Applications of Artificial Intelligence AAAI/IAAI 2002, Edmonton, Canada, pp. 273–278 (2002)Google Scholar
  46. 46.
    Mahadevan, S.: Samuel meets Amarel: Automating value function approximation using global state space analysis. In: Proceedings 20th National Conference on Artificial Intelligence and the 17th Innovative Applications of Artificial Intelligence Conference (AAAI 2005), Pittsburgh, US, pp. 1000–1005 (2005)Google Scholar
  47. 47.
    Mahadevan, S., Maggioni, M.: Proto-value functions: A Laplacian framework for learning representation and control in Markov decision processes. Journal of Machine Learning Research 8, 2169–2231 (2007)MathSciNetGoogle Scholar
  48. 48.
    Mannor, S., Rubinstein, R.Y., Gat, Y.: The cross-entropy method for fast policy search. In: Proceedings 20th International Conference on Machine Learning (ICML 2003), Washington, US, pp. 512–519 (2003)Google Scholar
  49. 49.
    Marbach, P., Tsitsiklis, J.N.: Approximate gradient methods in policy-space optimization of Markov reward processes. Discrete Event Dynamic Systems: Theory and Applications 13, 111–148 (2003)MATHCrossRefMathSciNetGoogle Scholar
  50. 50.
    McCallum, A.: Overcoming incomplete perception with utile distinction memory. In: Proceedings 10th International Conference on Machine Learning (ICML 1993), Amherst, US, pp. 190–196 (1993)Google Scholar
  51. 51.
    Menache, I., Mannor, S., Shimkin, N.: Basis function adaptation in temporal difference reinforcement learning. Annals of Operations Research 134, 215–238 (2005)MATHCrossRefMathSciNetGoogle Scholar
  52. 52.
    Millán, J.d.R., Posenato, D., Dedieu, E.: Continuous-action Q-learning. Machine Learning 49(2-3), 247–265 (2002)MATHCrossRefGoogle Scholar
  53. 53.
    Munos, R.: Finite-element methods with local triangulation refinement for continuous reinforcement learning problems. In: van Someren, M., Widmer, G. (eds.) ECML 1997. LNCS, vol. 1224, pp. 170–182. Springer, Heidelberg (1997)Google Scholar
  54. 54.
    Munos, R.: Policy gradient in continuous time. Journal of Machine Learning Research 7, 771–791 (2006)MathSciNetGoogle Scholar
  55. 55.
    Munos, R., Moore, A.: Variable-resolution discretization in optimal control. Machine Learning 49(2-3), 291–323 (2002)MATHCrossRefGoogle Scholar
  56. 56.
    Nakamura, Y., Moria, T., Satoc, M., Ishiia, S.: Reinforcement learning for a biped robot based on a CPG-actor-critic method. Neural Networks 20, 723–735 (2007)MATHCrossRefGoogle Scholar
  57. 57.
    Nedić, A., Bertsekas, D.P.: Least-squares policy evaluation algorithms with linear function approximation. Discrete Event Dynamic Systems 13, 79–110 (2003)MATHCrossRefMathSciNetGoogle Scholar
  58. 58.
    Ng, A.Y., Harada, D., Russell, S.: Policy invariance under reward transformations: Theory and application to reward shaping. In: Proceedings 16th International Conference on Machine Learning (ICML 1999), Bled, Slovenia, pp. 278–287 (1999)Google Scholar
  59. 59.
    Ng, A.Y., Jordan, M.I.: PEGASUS: A policy search method for large MDPs and POMDPs. In: Proceedings 16th Conference in Uncertainty in Artificial Intelligence (UAI 2000), Palo Alto, US, pp. 406–415 (2000)Google Scholar
  60. 60.
    Ormoneit, D., Sen, S.: Kernel-based reinforcement learning. Machine Learning 49(2-3), 161–178 (2002)MATHCrossRefGoogle Scholar
  61. 61.
    Pérez-Uribe, A.: Using a time-delay actor–critic neural architecture with dopamine-like reinforcement signal for learning in autonomous robots. In: Wermter, S., Austin, J., Willshaw, D.J. (eds.) Emergent Neural Computational Architectures Based on Neuroscience. LNCS (LNAI), vol. 2036, pp. 522–533. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  62. 62.
    Peters, J., Schaal, S.: Natural actor–critic. Neurocomputing 71(7-9), 1180–1190 (2008)CrossRefGoogle Scholar
  63. 63.
    Porta, J.M., Vlassis, N., Spaan, M.T., Poupart, P.: Point-based value iteration for continuous POMDPs. Journal of Machine Learning Research 7, 2329–2367 (2006)MathSciNetGoogle Scholar
  64. 64.
    Prokhorov, D., Wunsch, D.C.: Adaptive critic designs. IEEE Transactions on Neural Networks 8(5), 997–1007 (1997)CrossRefGoogle Scholar
  65. 65.
    Ratitch, B., Precup, D.: Sparse distributed memories for on-line value-based reinforcement learning. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 347–358. Springer, Heidelberg (2004)Google Scholar
  66. 66.
    Reynolds, S.I.: Adaptive resolution model-free reinforcement learning: Decision boundary partitioning. In: Proceedings 17th International Conference on Machine Learning (ICML 2000), Stanford University, US, pp. 783–790 (2000)Google Scholar
  67. 67.
    Riedmiller, M.: Neural fitted Q-iteration – first experiences with a data efficient neural reinforcement learning method. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS (LNAI), vol. 3720, pp. 317–328. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  68. 68.
    Riedmiller, M., Peters, J., Schaal, S.: Evaluation of policy gradient methods and variants on the cart-pole benchmark. In: Proceedings 2007 IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning (ADPRL 2007), Honolulu, US, pp. 254–261 (2007)Google Scholar
  69. 69.
    Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems. Tech. Rep. CUED/F-INFENG/TR166, Engineering Department, Cambridge University, UK (1994)Google Scholar
  70. 70.
    Santos, M.S., Vigo-Aguiar, J.: Analysis of a numerical dynamic programming algorithm applied to economic models. Econometrica 66(2), 409–426 (1998)MATHCrossRefMathSciNetGoogle Scholar
  71. 71.
    Singh, S.P., Jaakkola, T., Jordan, M.I.: Reinforcement learning with soft state aggregation. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems, vol. 7, pp. 361–368 (1995)Google Scholar
  72. 72.
    Sutton, R.S.: Learning to predict by the method of temporal differences. Machine Learning 3, 9–44 (1988)Google Scholar
  73. 73.
    Sutton, R.S.: Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: Proceedings 7th International Conference on Machine Learning (ICML 1990), Austin, US, pp. 216–224 (1990)Google Scholar
  74. 74.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  75. 75.
    Sutton, R.S., Barto, A.G., Williams, R.J.: Reinforcement learning is adaptive optimal control. IEEE Control Systems Magazine 12(2), 19–22 (1992)CrossRefGoogle Scholar
  76. 76.
    Sutton, R.S., McAllester, D.A., Singh, S.P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in Neural Information Processing Systems, vol. 12, pp. 1057–1063. MIT Press, Cambridge (2000)Google Scholar
  77. 77.
    Szepesvári, C., Smart, W.D.: Interpolation-based Q-learning. In: Proceedings 21st International Conference on Machine Learning (ICML 2004), Bannf, Canada, pp. 791–798 (2004)Google Scholar
  78. 78.
    Torczon, V.: On the convergence of pattern search algorithms. SIAM Journal on Optimization 7(1), 1–25 (1997)MATHCrossRefMathSciNetGoogle Scholar
  79. 79.
    Touzet, C.F.: Neural reinforcement learning for behaviour synthesis. Robotics and Autonomous Systems 22(3-4), 251–281 (1997)CrossRefGoogle Scholar
  80. 80.
    Tsitsiklis, J.N., Van Roy, B.: Feature-based methods for large scale dynamic programming. Machine Learning 22(1-3), 59–94 (1996)MATHCrossRefGoogle Scholar
  81. 81.
    Tsitsiklis, J.N., Van Roy, B.: An analysis of temporal difference learning with function approximation. IEEE Transactions on Automatic Control 42(5), 674–690 (1997)MATHCrossRefGoogle Scholar
  82. 82.
    Uther, W.T.B., Veloso, M.M.: Tree based discretization for continuous state space reinforcement learning. In: Proceedings 15th National Conference on Artificial Intelligence and 10th Innovative Applications of Artificial Intelligence Conference (AAAI 1998/IAAI 1998), Madison, US, pp. 769–774 (1998)Google Scholar
  83. 83.
    Vrabie, D., Pastravanu, O., Abu-Khalaf, M., Lewis, F.: Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica 45(2), 477–484 (2009)MATHCrossRefMathSciNetGoogle Scholar
  84. 84.
    Waldock, A., Carse, B.: Fuzzy Q-learning with an adaptive representation. In: Proceedings 2008 IEEE World Congress on Computational Intelligence (WCCI 2008), Hong Kong, pp. 720–725 (2008)Google Scholar
  85. 85.
    Wang, X., Tian, X., Cheng, Y.: Value approximation with least squares support vector machine in reinforcement learning system. Journal of Computational and Theoretical Nanoscience 4(7-8), 1290–1294 (2007)CrossRefGoogle Scholar
  86. 86.
    Watkins, C.J.C.H.: Learning from delayed rewards. Ph.D. thesis, King’s College, Oxford (1989)Google Scholar
  87. 87.
    Watkins, C.J.C.H., Dayan, P.: Q-learning. Machine Learning 8, 279–292 (1992)MATHGoogle Scholar
  88. 88.
    Wiering, M.: Convergence and divergence in standard and averaging reinforcement learning. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 477–488. Springer, Heidelberg (2004)Google Scholar
  89. 89.
    Williams, R.J., Baird, L.C.: Tight performance bounds on greedy policies based on imperfect value functions. In: Proceedings 8th Yale Workshop on Adaptive and Learning Systems, New Haven, US, pp. 108–113 (1994)Google Scholar
  90. 90.
    Xu, X., Hu, D., Lu, X.: Kernel-based least-squares policy iteration for reinforcement learning. IEEE Transactions on Neural Networks 18(4), 973–992 (2007)CrossRefGoogle Scholar
  91. 91.
    Yu, H., Bertsekas, D.P.: Convergence results for some temporal difference methods based on least-squares. Tech. Rep. LIDS 2697, Massachusetts Institute of Technology, Cambridge, US (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Lucian Buşoniu
    • 1
  • Bart De Schutter
    • 2
  • Robert Babuška
    • 2
  1. 1.Delft Center for Systems and ControlDelft University of TechnologyDelftThe Netherlands
  2. 2.Delft Center for Systems and Control & Marine and Transport Technology DepartmentDelft University of TechnologyDelftThe Netherlands

Personalised recommendations