On Affect and Self-adaptation: Potential Benefits of Valence-Controlled Action-Selection

  • Joost Broekens
  • Walter A. Kosters
  • Fons J. Verbeek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4527)


Psychological studies have shown that emotion and affect influence learning. We employ these findings in a machine-learning meta-parameter context, and dynamically couple an adaptive agent’s artificial affect to its action-selection mechanism (Boltzmann β). The agent’s performance on two important learning problems is measured. The first consists of learning to cope with two alternating goals. The second consists of learning to prefer a later larger reward (global optimum) for an earlier smaller one (local optimum). Results show that, compared to several control conditions, coupling positive affect to exploitation and negative affect to exploration has several important benefits. In the alternating-goal task, it significantly reduces the agent’s “goal-switch search peak”. The agent finds its new goal faster. In the second task, artificial affect facilitates convergence to a global instead of a local optimum, while permitting to exploit that local optimum. We conclude that affect-controlled action-selection has adaptation benefits.


Local Optimum Reinforcement Learning Markov Decision Process Goal Location Large Reward 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ashby, F.G., Isen, A.M., Turken, U.: A neuropsychological theory of positive affect and its influence on cognition. Psychological Review 106, 529–550 (1999)CrossRefGoogle Scholar
  2. 2.
    Damasio, A.R.: Descartes’ error: Emotion, reason, and the human brain. Gosset/Putnam Press, New York (1994)Google Scholar
  3. 3.
    Forgas, J.P.: Feeling is believing? The role of processing strategies in mediating affective influences in beliefs. In: Frijda, N., et al. (eds.) Emotions and Beliefs, pp. 108–143. Cambridge University Press, Cambridge (2000)Google Scholar
  4. 4.
    Phaf, R.H., Rotteveel, M.: Affective modulation of recognition bias. Emotion 15, 309–318 (2005)CrossRefGoogle Scholar
  5. 5.
    Russell, J.A.: Core affect and the psychological construction of emotion. Psychological Review 110, 145–172 (2003)CrossRefGoogle Scholar
  6. 6.
    Dreisbach, G., Goschke, T.: How positive affect modulates cognitive control: Reduced perseveration at the cost of increased distractibility. Journal of Experimental Psychology 30, 343–353 (2004)Google Scholar
  7. 7.
    Rose, S.A., Futterweit, L.R., Jankowski, J.J.: The relation of affect to attention and learning in infancy. Child Development 70, 549–559 (1999)CrossRefGoogle Scholar
  8. 8.
    Craig, S.D., Graesser, A.C., Sullins, J., Gholson, B.: Affect and learning: An exploratory look into the role of affect in learning with Autotutor. Journal of Educational Media 29, 241–250 (2004)Google Scholar
  9. 9.
    Sutton, R., Barto, A.: Reinforcement learning, An introduction. MIT Press, Cambridge (1998)Google Scholar
  10. 10.
    Csikszentmihalyi, M.: Flow: The psychology of optimal experience. Harper Row, New York (1990)Google Scholar
  11. 11.
    Custers, R., Aarts, H.: Positive affect as implicit motivator: On the nonconscious operation of behavioral goals. Journal of Personality and Social Psychology 89, 129–142 (2005)CrossRefGoogle Scholar
  12. 12.
    Doya, K.: Metalearning and neuromodulation. Neural Networks 15, 495–506 (2002)CrossRefGoogle Scholar
  13. 13.
    Tyrrell, T.: Computational mechanisms for action selection, PhD thesis. University of Edinburgh (1993)Google Scholar
  14. 14.
    Avila-Garcia, O., Cañamero, L.: Using hormonal feedback to modulate action selection in a competitive scenario. In: Proceedings of the 8th Intl. Conf. on Simulation of Adaptive Behavior, pp. 243–252 (2004)Google Scholar
  15. 15.
    Belavkin, R.V.: On relation between emotion and entropy. In: Proceedings of the AISB’04 Symposium on Emotion, Cognition and Affective Computing, Leeds, UK, pp. 1–8 (2004)Google Scholar
  16. 16.
    Botelho, L.M., Coelho, H.: Information processing, motivation and decision making. In: Proc. 4th International Workshop on Artificial Intelligence in Economics and Management (1998)Google Scholar
  17. 17.
    Schweighofer, N., Doya, K.: Meta-learning in reinforcement learning. Neural Networks 16, 5–9 (2003)CrossRefGoogle Scholar
  18. 18.
    Hogewoning, E., Broekens, J., Eggermont, J., Bovenkamp, E.G.P.: Strategies for affect-controlled action-selection in Soar-RL. In: Mira, J., Álvarez, J.R. (eds.) IWINAC 2007. LNCS, vol. 4528, pp. 501–510. Springer, Heidelberg (2007)CrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Joost Broekens
    • 1
  • Walter A. Kosters
    • 1
  • Fons J. Verbeek
    • 1
  1. 1.Leiden Institute of Advanced Computer Science, Leiden University, P.O. Box 9500, 2300 RA LeidenThe Netherlands

Personalised recommendations