Abstract
We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Andre, D., Russell, S.: Programmable reinforcement learning agents. In: NIPS (2001)
Clouse, J., Utgoff, P.: A teaching method for reinforcement learning. In: Proc. ICML 1992 (1992)
Gordon, D., Subramanian, D.: A multistrategy learning scheme for agent knowledge acquisition. Informatica 17, 331–346 (1994)
Kuhlmann, G., Stone, P., Mooney, R., Shavlik, J.: Guiding a reinforcement learner with natural language advice: Initial results in RoboCup soccer. In: AAAI Workshop on Supervisory Control of Learning and Adaptive Systems (2004)
Laud, A., DeJong, G.: Reinforcement learning and shaping: Encouraging intended behaviors. In: ICML (2002)
Lin, L.: Self-improving reactive agents based on reinforcement learning, planning, and teaching. Machine Learning 8, 293–321 (1992)
Maclin, R., Shavlik, J.: Creating advice-taking reinforcement learners. Machine Learning 22, 251–281 (1996)
Maclin, R., Shavlik, J., Torrey, L., Walker, T.: Knowledge-based support vector regression for reinforcement learning. In: IJCAI Workshop on Reasoning, Representation, and Learning in Computer Games (2005)
Maclin, R., Shavlik, J., Torrey, L., Walker, T., Wild, E.: Giving advice about preferred actions to reinforcement learners via knowledge-based kernel regression. In: AAAI (2005)
Mangasarian, O., Shavlik, J., Wild, E.: Knowledge-based kernel approximation. JMLR 5, 1127–1141 (2004)
Noda, I., Matsubara, H., Hiraki, K., Frank, I.: Soccer server: A tool for research on multiagent systems. Applied Artificial Intelligence 12, 233–250 (1998)
Price, B., Boutilier, C.: Implicit imitation in multiagent reinforcement learning. In: ICML (1999)
Selfridge, O., Sutton, R., Barto, A.: Training and tracking in robotics. In: IJCAI (1985)
Sherstov, A., Stone, P.: Improving action selection in MDP’s via knowledge transfer. In: AAAI (2005)
Singh, S.: Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning 8(3-4), 323–339 (1992)
Stone, P., Sutton, R.: Scaling reinforcement learning toward RoboCup soccer. In: ICML (2001)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Taylor, M., Stone, P.: Behavior transfer for value-function-based reinforcement learning. In: 4th Int. Joint Conf. on Autonomous Agents and Multiagent Sys. (2005)
Thrun, S., Mitchell, T.: Learning one more thing. In: IJCAI (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Torrey, L., Walker, T., Shavlik, J., Maclin, R. (2005). Using Advice to Transfer Knowledge Acquired in One Reinforcement Learning Task to Another. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds) Machine Learning: ECML 2005. ECML 2005. Lecture Notes in Computer Science(), vol 3720. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11564096_40
Download citation
DOI: https://doi.org/10.1007/11564096_40
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29243-2
Online ISBN: 978-3-540-31692-3
eBook Packages: Computer ScienceComputer Science (R0)