An Adaptive Robot Motivational System

  • George Konidaris
  • Andrew Barto
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4095)


We present a robot motivational system design framework. The framework represents the underlying (possibly conflicting) goals of the robot as a set of drives, while ensuring comparable drive levels and providing a mechanism for drive priority adaptation during the robot’s lifetime. The resulting drive reward signals are compatible with existing reinforcement learning methods for balancing multiple reward functions. We illustrate the framework with an experiment that demonstrates some of its benefits.


Motivational System Action Selection Priority Level Reward Function Autonomous Robot 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bersini, H.: Reinforcement learning for homeostatic endogenous variables. In: From Animals to Animats 3: Proceedings of the Third International Conference on the Simulation of Adaptive Behavior, pp. 325–333 (1994)Google Scholar
  2. 2.
    Blumberg, B.M., Todd, P.M., Maes, P.: No bad dogs: Ethological lessons for learning in hamsterdam. In: From Animals to Animats 4: Proceedings of the Fourth International Conference on the Simulation of Adaptive Behavior, pp. 295–304 (1996)Google Scholar
  3. 3.
    Brooks, R.A.: The role of learning in autonomous robots. In: Proceedings of the Fourth Annual Workshop on Computational Learning Theory (COLT 1991), pp. 5–10 (1991)Google Scholar
  4. 4.
    Cañamero, L.: Modeling motivations and emotions as a basis for intelligent behavior. In: Lewis Johnson, W. (ed.) Proceedings of the First International Conference on Autonomous Agents, pp. 148–155. ACM Press, New York (1997)CrossRefGoogle Scholar
  5. 5.
    Cos-Aguilera, I., Cañamero, L., Hayes, G.M.: Motivation-driven learning of object affordances: First experiments using a simulated khepera robot. In: Detjer, F., Dörner, D., Schaub, H. (eds.) The Logic of Cognitive Systems: Proceedings of the Fifth International Conference on Cognitive Modeling (ICCM 2003), pp. 57–62 (2003)Google Scholar
  6. 6.
    Doya, K., Uchibe, E.: The cyber rodent project: Exploration of adaptive mechanisms for self-preservation and self-reproduction. Adaptive Behavior 13(2), 149–160 (2005)CrossRefGoogle Scholar
  7. 7.
    Halperin, J.R.P.: Machine motivation. In: From Animals to Animats: Proceedings of the First International Conference on the Simulation of Adaptive Behavior, pp. 213–221 (1990)Google Scholar
  8. 8.
    Humphrys, M.: Action selection methods using reinforcement learning. In: From Animats to Animats 4: Proceedings of the Fourth International Conference on the Simulation of Adaptive Behavior, pp. 135–144 (1996)Google Scholar
  9. 9.
    Konidaris, G.D., Hayes, G.M.: Anticipatory learning for focusing search in reinforcement learning agents. In: The Second Workshop on Anticipatory Behavior in Adaptive Learning Systems (July 2004)Google Scholar
  10. 10.
    Konidaris, G.D., Hayes, G.M.: Estimating future reward in reinforcement learning animats using associative learning. In: From Animals to Animats 8: Proceedings of the 8th International Conference on the Simulation of Adaptive Behavior, pp. 297–304 (2004)Google Scholar
  11. 11.
    Konidaris, G.D., Hayes, G.M.: An architecture of behavior-based reinforcement learning. Adaptive Behavior 13(1), 5–32 (2005)CrossRefGoogle Scholar
  12. 12.
    McFarland, D., Bosser, T.: Intelligent Behavior in Animals and Robots. MIT Press, Cambridge (1994)Google Scholar
  13. 13.
    Savage, T.: Artificial motives: a review of motivation in artificial creatures. Connection Science 12(3/4), 211–277 (2000)CrossRefGoogle Scholar
  14. 14.
    Spier, E.: From Reactive Behaviour to Adaptive Behaviour: Motivational Models for Behaviour in Animals and Robots. PhD thesis, Balliol College, University of Oxford (1997)Google Scholar
  15. 15.
    Spier, E., McFarland, D.: Possibly optimal decision-making under self-sufficiency and autonomy. Journal of Theoretical Biology 189(3), 317–331 (1997)CrossRefGoogle Scholar
  16. 16.
    Sprague, N., Ballard, D.H.: Multiple-goal reinforcement learning with modular Sarsa(0). In: Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI 2003), pp. 1445–1447 (2003)Google Scholar
  17. 17.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  18. 18.
    Toates, F.: Motivational Systems, ch. 4. Cambridge University Press, Cambridge (1986)Google Scholar
  19. 19.
    Whitehead, S., Karlsson, J., Tenenberg, J.: Learning multiple goal behavior via task decomposition and dynamic policy merging. In: Connell, J.H., Mahadevan, S. (eds.) Robot Learning, pp. 45–78. Kluwer Academic Publishers, Dordrecht (1992)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • George Konidaris
    • 1
  • Andrew Barto
    • 1
  1. 1.Autonomous Learning Laboratory, Department of Computer ScienceUniversity of Massachusetts at Amherst 

Personalised recommendations