Incremental Skill Acquisition for Self-motivated Learning Animats

  • Andrea Bonarini
  • Alessandro Lazaric
  • Marcello Restelli
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4095)


A central role in the development process of children is played by self-exploratory activities. Through a playful interaction with the surrounding environment, they test their own capabilities, explore novel situations, and understand how their actions affect the world. During this kind of exploration, interesting situations may be discovered. By learning to reach these situations, a child incrementally develops more and more complex skills. Inspired by studies from psychology, neuroscience, and machine learning, we designed SMILe (Self-Motivated Incremental Learning), a learning framework that allows artificial agents to autonomously identify and learn a set of abilities useful to face several different tasks, through an iterated three phase process: by means of a random exploration of the environment (babbling phase), the agent identifies interesting situations and generates an intrinsic motivation (motivating phase) aimed at learning how to get into these situations (skill acquisition phase). This process incrementally increases the skills of the agent, so that new interesting configurations can be experienced. We present results on two gridworld environments to show how SMILe makes it possible to learn skills that enable the agent to perform well and robustly in many different tasks.


Intrinsic Motivation Reinforcement Learn Motivate Learning Skill Acquisition Reward Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barto, A.G., Mahadevan, S.: Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems 13(4), 341–379 (2003)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Barto, A.G., Singh, S., Chentanez, N.: Intrinsically motivated learning of hierarchical collections of skills. In: Proceedings of ICDL (2004)Google Scholar
  3. 3.
    Berlyne, D.E.: Conflict, Arousal, and Curiosity. McGraw-Hill, New York (1960)CrossRefGoogle Scholar
  4. 4.
    Bonarini, A., Lazaric, A., Restelli, M.: Smile: Self-motivated incremental learning. Technical report, Politecnico di Milano (2006),
  5. 5.
    Kakade, S., Dayan, P.: Dopamine: Generalization and bonuses. Neural Networks 15, 549–559 (2002)CrossRefGoogle Scholar
  6. 6.
    Konidaris, G.D., Hayes, G.M.: An architecture for behavior-based reinforcement learning. Adaptive Behavior 13(1), 5–32 (2005)CrossRefGoogle Scholar
  7. 7.
    Lungarella, M., Metta, G., Pfeifer, R., Sandini, C.: Developmental robotics: a survey. Connection Science 15(4), 151–190 (2003)CrossRefGoogle Scholar
  8. 8.
    Marshall, J., Blank, D., Meeden, L.: An emergent framework for self-motivation in developmental robotics. In: Proceedings of ICDL (2004)Google Scholar
  9. 9.
    McGovern, A., Barto, A.G.: Automatic discovery of subgoals in reinforcement learning using diverse density. In: Proceedings of ICML (2001)Google Scholar
  10. 10.
    Meltzoff, A., Moore, M.: Explaining facial imitation: a theoretical model. Early Development and Parenting 6, 179–192 (1997)CrossRefGoogle Scholar
  11. 11.
    Menache, I., Mannor, S., Shimkin, N.: Q-Cut - Dynamic Discovery of Sub-goals in Reinforcement Learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS, vol. 2430, p. 295. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  12. 12.
    Oudeyer, P.-Y., Kaplan, F., Hafner, V.: The playground experiment: Task-independent development of a curious robot. In: AAAI Spring Symposium Workshop on Developmental Robotics (2005)Google Scholar
  13. 13.
    Piaget, J.: The Origins of Intelligence in Children. Norton, New York (1952)CrossRefGoogle Scholar
  14. 14.
    Ratitch, B., Precup, D.: Using mdp characteristics to guide exploration in reinforcement learning. In: European Conference on Reinforcement Learning (2003)Google Scholar
  15. 15.
    Schmidhuber, J.: Self-motivated development through rewards for predictor errors / improvements. In: AAAI Spring Symposium on Developmental Robotics (2005)Google Scholar
  16. 16.
    Stout, A., Konidaris, G., Barto, A.: Intrinsically motivated reinforcement learning: A promising framework for developmental robot learning. In: AAAI Spring Symposium on Developmental Robotics (2005)Google Scholar
  17. 17.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  18. 18.
    Sutton, R.S., Precup, D., Singh, S.P.: Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112, 181–211 (1999)MATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Uchibe, E., Doya, K.: Reinforcement learning with multiple heterogeneous modules: A framework for developmental robot learning. In: Proceedings of ICDL (2005)Google Scholar
  20. 20.
    Weng, J., McClelland, A., Sporns, O., Stockman, I., Sur, M., Thelen, E.: Autonomous mental development by robots and animals. Science 291, 599–600 (2001)CrossRefGoogle Scholar
  21. 21.
    Weng, J., Zhang, Y.: Novelty and reinforcement learning in the value system of developmental robots. In: International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Andrea Bonarini
    • 1
  • Alessandro Lazaric
    • 1
  • Marcello Restelli
    • 1
  1. 1.Department of Electronics and InformaticsPolitecnico di MilanoMilanItaly

Personalised recommendations