Transferring Evolved Reservoir Features in Reinforcement Learning Tasks
The major goal of transfer learning is to transfer knowledge acquired on a source task in order to facilitate learning on another, different, but usually related, target task. In this paper, we are using neuroevolution to evolve echo state networks on the source task and transfer the best performing reservoirs to be used as initial population on the target task. The idea is that any non-linear, temporal features, represented by the neurons of the reservoir and evolved on the source task, along with reservoir properties, will be a good starting point for a stochastic search on the target task. In a step towards full autonomy and by taking advantage of the random and fully connected nature of echo state networks, we examine a transfer method that renders any inter-task mappings of states and actions unnecessary. We tested our approach and that of inter-task mappings in two RL testbeds: the mountain car and the server job scheduling domains. Under various setups the results we obtained in both cases are promising.
Unable to display preview. Download preview PDF.
- 1.Bahceci, E., Miikkulainen, R.: Transfer of evolved pattern-based heuristics in games. In: IEEE Symposium on Computational Intelligence and Games (2008)Google Scholar
- 2.Chatzidimitriou, K.C., Mitkas, P.A.: A neat way for evolving echo state networks. In: European Conference on Artificial Intelligence, pp. 909–914 (2010)Google Scholar
- 3.Fernández, F., Veloso, M.: Probabilistic policy reuse in a reinforcement learning agent. In: 5th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 720–727 (2006)Google Scholar
- 4.Jaeger, H.: Tutorial on training recurrent neural networks, covering BPTT, RTRL, EKF and the “ echo state network” approach. Tech. Rep. GMD Report 159, German National Research Center for Information Technology (2002)Google Scholar
- 8.Stone, P.: Learning and multiagent reasoning for autonomous agents. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 13–30 (January 2007), http://www.ijcai-07.org/
- 9.Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
- 11.Tanner, B., White, A.: Rl-glue: Language-independent software for reinforcement-learning experiments. Journal of Machine Learning Research 10, 2133–2136 (2010)Google Scholar
- 14.Taylor, M.E., Kuhlmann, G., Stone, P.: Autonomous transfer for reinforcement learning. In: AAMAS 2008: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 283–290 (2008)Google Scholar
- 15.Taylor, M.E., Whiteson, S., Stone, P.: Transfer via inter-task mappings in policy search reinforcement learning. In: 6th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 1–8 (2007)Google Scholar