Advertisement

Gradient Based Algorithms with Loss Functions and Kernels for Improved On-Policy Control

  • Matthew Robards
  • Peter Sunehag
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7188)

Abstract

We introduce and empirically evaluate two novel online gradient-based reinforcement learning algorithms with function approximation – one model based, and the other model free. These algorithms come with the possibility of having non-squared loss functions which is novel in reinforcement learning, and seems to come with empirical advantages. We further extend a previous gradient based algorithm to the case of full control, by using generalized policy iteration. Theoretical properties of these algorithms are studied in a companion paper.

Keywords

Loss Function Gradient Descent Markov Decision Process Reproduce Kernel Hilbert Space Noisy Observation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Baird, L., Moore, A.: Gradient descent for general reinforcement learning. In: Neural Information Processing Systems, vol. 11, pp. 968–974. MIT Press (1998)Google Scholar
  2. 2.
    Engel, Y., Mannor, S., Meir, R.: Reinforcement learning with Gaussian processes. In: 22nd International Conference on Machine Learning (ICML 2005), Bonn, Germany, pp. 201–208 (2005)Google Scholar
  3. 3.
    Engel, Y., Mannor, S., Meir, R.: Bayes meets bellman: The gaussian process approach to temporal difference learning. In: Proc. of the 20th International Conference on Machine Learning, pp. 154–161 (2003)Google Scholar
  4. 4.
    Maei, H., Szepesvri, C., Bhatnagar, S., Sutton, R.: Toward off-policy learning control with function approximation. In: Proceedings of the 27th International Conference on Machine Learning (2010)Google Scholar
  5. 5.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)zbMATHGoogle Scholar
  6. 6.
    Robards, M., Sunehag, P.: Online convex reinforcement learning. In: Submitted to 9th EWRL (2011)Google Scholar
  7. 7.
    Robards, M., Sunehag, P., Sanner, S., Marthi, B.: Sparse Kernel-SARSA(λ) with an Eligibility Trace. In: Gunopulos, D., Hofmann, T., Malerba, D., Vazirgiannis, M. (eds.) ECML PKDD 2011. LNCS, vol. 6913, pp. 1–17. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  8. 8.
    Sutton, R., Barto, A.: Reinforcement Learning. The MIT Press (1998)Google Scholar
  9. 9.
    Sutton, R., Maei, H., Precup, D., Bhatnagar, S., Silver, D., Szepesvri, C., Wiewiora, E.: Fast gradient-descent methods for temporal-difference learning with linear function approximation. In: Proceedings of the 26th International Conference on Machine Learning (2009)Google Scholar
  10. 10.
    Sutton, R., Szepesvári, C., Maei, H.: A convergent o(n) temporal-difference algorithm for off-policy learning with linear function approximation. In: NIPS, pp. 1609–1616. MIT Press (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Matthew Robards
    • 1
  • Peter Sunehag
    • 1
  1. 1.Australian National UniversityNictaAustralia

Personalised recommendations