Enhanced Temporal Difference Learning Using Compiled Eligibility Traces
Eligibility traces have been shown to substantially improve the convergence speed of temporal difference learning algorithms, by maintaining a record of recently experienced states. This paper presents an extension of conventional eligibility traces (compiled traces) which retain additional information about the agent’s experience within the environment. Empirical results show that compiled traces outperform conventional traces when applied to policy evaluation tasks using a tabular representation of the state values.
KeywordsReinforcement Learning Previous Episode Policy Iteration Current Episode Eligibility Trace
Unable to display preview. Download preview PDF.
- 1.Sutton, R.S.: Learning to predict by the methods of temporal differences. Machine Learning 3, 9–44 (1988)Google Scholar
- 3.Rummery, G., Niranjan, M.: On-line Q-Learning Using Connectionist Systems. Cambridge University Engineering Department, Cambridge (1994)Google Scholar
- 4.Watkins, C.J.C.H.: Learning from Delayed Rewards, PhD Thesis, Cambridge University (1989)Google Scholar
- 6.Kaelbling, L.P.: Learning to Achieve Goals. In: Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, Chambéry, France (1993)Google Scholar
- 7.Ollington, R., Vamplew, P.: Concurrent Q-Learning for Autonomous Mapping and Navigation. In: The 2nd International Conference on Computational Intelligence, Robotics and Autonomous Systems, Singapore (2003)Google Scholar
- 9.Cichosz, P.: Truncating temporal differences: On the efficient implementation of TD(λ) for reinforcement learning. Journal of Artificial Intelligence Research 2, 287–318 (1995)Google Scholar
- 10.Sutton, R.S.: Generalisation in reinforcement learning: Successful examples using sparse coarse coding. In: Touretzky, D.S., Mozer, M.C., Hasselmo, M.E. (eds.) Advances in Neural Information Processing Systems: Proceedings of the 1995 Conference, pp. 1038–1044. The MIT Press, Cambridge (1996)Google Scholar
- 11.Kretchmar, R.M., Anderson, C.W.: Comparison of CMACs and RBFs for local function approximators in reinforcement learning. In: IEEE International Conference on Neural Networks (1997)Google Scholar