Motivational Engine with Sub-goal Identification in Neuroevolution Based Cognitive Robotics

  • Rodrigo Salgado
  • Abraham Prieto
  • Pilar Caamaño
  • Francisco Bellas
  • Richard J. DuroEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9648)


A first approach towards a new motivational system for an autonomous robot that can learn chains of sub-goals leading to a final reward is proposed in this paper. The motivational system provides the motivation that guides the robot operation according to its knowledge of its sensorial space so that rewards are maximized during its lifetime. In order to do this, a motivational engine progressively and interactively creates an internal model of expected future reward (value function) for areas of the robot’s state space, through a neuroevolutionary process, over samples obtained in the sensorial (state space) traces followed by the robot whenever it obtained a reward. To improve this modelling process, a strategy is proposed to decompose the global value function leading to the reward or goal into several more local ones, thus discovering sub-goals that simplify the whole learning process and that can be reused in the future. The motivational engine is tested in a simulated experiment with very promising results.


Neuroevolution Cognitive systems Motivation Autonomous robots 



This work was partially funded by the EU’s H2020 research and innovation programme under grant agreement No 640891 (DREAM project) and by the Xunta de Galicia and European Regional Development Funds under grants GRC 2013-050 and redTEIC network (R2014/037).


  1. 1.
    Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp. Educ. Psychol. 25, 54–67 (2000)CrossRefGoogle Scholar
  2. 2.
    Scott, P.D., Markovitch, S.: Learning novel domains through curiosity and conjecture. In: Proceedings IJCAI 1989, pp. 669–674 (1989)Google Scholar
  3. 3.
    Shmidhuber, J.: Adaptive confidence and adaptive curiosity. Technical Report. Institut fur Informatik, Technische Universität Munchen (1991)Google Scholar
  4. 4.
    Lenat, D.B.: AM: an artificial intelligence approach to discovery in mathematics as heuristic search, Doctoral Dissertation No. STAN-CS-76-570, Department of Computer Science, Stanford University (1976)Google Scholar
  5. 5.
    Sutton, R.S.: Reinforcement learning architectures for animals. In: From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior, pp. 288–296 (1991)Google Scholar
  6. 6.
    Oudeyer, P.Y., Kaplan, F.: What is intrinsic motivation? a typology of computational approaches. Front. Neurorobot. 1(A.6), 1–14 (2007)Google Scholar
  7. 7.
    McGovern, A., Barto, A.: Automatic discovery of subgoals in reinforcement learning using diverse density. Technical report of the faculty publication series, Computer Science Department, University of Massachusetts, Amherst (2001)Google Scholar
  8. 8.
    Menache, I., Mannor, S., Shimkin, N.: Q-cut - dynamic discovery of sub-goals in reinforcement learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 295–306. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Simsek, O., Barto, A.: Betweenness centrality as a basis for forming skills. Technical report, Department of Computer Science, University of Massachusetts (2007)Google Scholar
  10. 10.
    Mannor, S., Menache, I., Hoze, A., Klein, U.: Dynamic abstraction in reinforcement learning via clustering. In: Proceedings of the Twenty-First International Conference on Machine Learning, pp. 560–567. ACM (2004)Google Scholar
  11. 11.
    Konidaris, G., Barto, A.: Sensorimotor abstraction selection for efficient, autonomous robot skill acquisition. In: 7th IEEE International Conference on Development and Learning, ICDL 2008, pp. 151–156, (2008)Google Scholar
  12. 12.
    Salge, C., Glackin, C., Polani, D.: Approximation of empowerment in the continuous domain. Adv. Complex Syst. 16(1–2), 1250079 (2013)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Bellas, F., Duro, R.J., Faina, A., Souto, D.: Multilevel Darwinist Brain (MDB): artificial evolution in a cognitive architecture for real robots. IEEE Trans. Auton. Ment. Dev. 2(4), 340–354 (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Rodrigo Salgado
    • 1
  • Abraham Prieto
    • 1
  • Pilar Caamaño
    • 1
  • Francisco Bellas
    • 1
  • Richard J. Duro
    • 1
    Email author
  1. 1.Integrated Group for Engineering ResearchUniversidade da CoruñaFerrolSpain

Personalised recommendations