Transferring Evolved Reservoir Features in Reinforcement Learning Tasks

  • Kyriakos C. Chatzidimitriou
  • Ioannis Partalas
  • Pericles A. Mitkas
  • Ioannis Vlahavas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7188)


The major goal of transfer learning is to transfer knowledge acquired on a source task in order to facilitate learning on another, different, but usually related, target task. In this paper, we are using neuroevolution to evolve echo state networks on the source task and transfer the best performing reservoirs to be used as initial population on the target task. The idea is that any non-linear, temporal features, represented by the neurons of the reservoir and evolved on the source task, along with reservoir properties, will be a good starting point for a stochastic search on the target task. In a step towards full autonomy and by taking advantage of the random and fully connected nature of echo state networks, we examine a transfer method that renders any inter-task mappings of states and actions unnecessary. We tested our approach and that of inter-task mappings in two RL testbeds: the mountain car and the server job scheduling domains. Under various setups the results we obtained in both cases are promising.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bahceci, E., Miikkulainen, R.: Transfer of evolved pattern-based heuristics in games. In: IEEE Symposium on Computational Intelligence and Games (2008)Google Scholar
  2. 2.
    Chatzidimitriou, K.C., Mitkas, P.A.: A neat way for evolving echo state networks. In: European Conference on Artificial Intelligence, pp. 909–914 (2010)Google Scholar
  3. 3.
    Fernández, F., Veloso, M.: Probabilistic policy reuse in a reinforcement learning agent. In: 5th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 720–727 (2006)Google Scholar
  4. 4.
    Jaeger, H.: Tutorial on training recurrent neural networks, covering BPTT, RTRL, EKF and the “ echo state network” approach. Tech. Rep. GMD Report 159, German National Research Center for Information Technology (2002)Google Scholar
  5. 5.
    Madden, M.G., Howley, T.: Transfer of experience between reinforcement learning environments with progressive difficulty. Artif. Intell. Rev. 21(3-4), 375–398 (2004)zbMATHCrossRefGoogle Scholar
  6. 6.
    Singh, S.P., Sutton, R.S.: Reinforcement learning with replacing eligibility traces. Machine Learning 22(1-3), 123–158 (1996)zbMATHCrossRefGoogle Scholar
  7. 7.
    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary Computation 10(2), 99–127 (2002)CrossRefGoogle Scholar
  8. 8.
    Stone, P.: Learning and multiagent reasoning for autonomous agents. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 13–30 (January 2007),
  9. 9.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  10. 10.
    Szita, I., Gyenes, V., Lőrincz, A.: Reinforcement Learning with Echo State Networks. In: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (eds.) ICANN 2006. LNCS, vol. 4131, pp. 830–839. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Tanner, B., White, A.: Rl-glue: Language-independent software for reinforcement-learning experiments. Journal of Machine Learning Research 10, 2133–2136 (2010)Google Scholar
  12. 12.
    Taylor, M., Stone, P.: Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research 10, 1633–1685 (2009)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Taylor, M.E., Jong, N.K., Stone, P.: Transferring Instances for Model-Based Reinforcement Learning. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part II. LNCS (LNAI), vol. 5212, pp. 488–505. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Taylor, M.E., Kuhlmann, G., Stone, P.: Autonomous transfer for reinforcement learning. In: AAMAS 2008: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 283–290 (2008)Google Scholar
  15. 15.
    Taylor, M.E., Whiteson, S., Stone, P.: Transfer via inter-task mappings in policy search reinforcement learning. In: 6th International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 1–8 (2007)Google Scholar
  16. 16.
    Torrey, L., Shavlik, J., Walker, T., Maclin, R.: Skill Acquisition Via Transfer Learning and Advice Taking. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS (LNAI), vol. 4212, pp. 425–436. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  17. 17.
    Whiteson, S., Stone, P.: Evolutionary function approximation for reinforcement learning. Journal of Machine Learning Research 7, 877–917 (2006)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Kyriakos C. Chatzidimitriou
    • 1
    • 2
  • Ioannis Partalas
    • 3
  • Pericles A. Mitkas
    • 1
    • 2
  • Ioannis Vlahavas
    • 3
  1. 1.Dept. of Electrical & Computer EngineeringAristotle University of ThessalonikiGreece
  2. 2.Informatics and Telematics InstituteCentre for Research and TechnologyHellasGreece
  3. 3.Dept. of InformaticsAristotle University of ThessalonikiGreece

Personalised recommendations