Producing Parameterized Value Functions Through Modulation for Cognitive Developmental Robots

  • Alejandro Romero
  • Francisco BellasEmail author
  • Jose A. Becerra
  • Richard J. Duro
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1093)


Parameterizing value functions as a representation of robotic tasks in different domains allows for their generalization, and can provide a way to transfer knowledge to new situations. To this end, in this paper we propose a modulation based mechanism embedded within a cognitive architecture for robots. It makes use of the combined operation of the long-term memory and the motivational system in order to select candidate primitive value functions for transfer. These are then adapted to the new situation through the addition of modulatory ANNs to progressively conform new parameterized value functions able to address more complex situations in a developmental manner. The proposed method is tested in a Baxter robot, which must solve different tasks in a cooking setup.


Developmental robotics Motivational system Value functions Long-term memory Cognitive architecture 



This work has been partially funded by the EU’s H2020 research programme (grant No 640891 DREAM), Ministerio de Ciencia, Innovación y Universidades of Spain/FEDER (grant RTI2018-101114-B-I00), Xunta de Galicia and FEDER (grant ED431C 2017/12), and by the Spanish Ministry of Education, Culture and Sports for the FPU grant of Alejandro Romero.


  1. 1.
    Doncieux, S., et al.: Open-ended learning: a conceptual framework based on representational redescription. Front. Neurorobotics 12, 59 (2018)CrossRefGoogle Scholar
  2. 2.
    Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Lazaric, A.: Transfer in reinforcement learning: a framework and a survey, adaptation, learning, and optimization (2012)Google Scholar
  4. 4.
    Zhao, C., Hospedales, T. M., Stulp, F., Sigaud, O.: Tensor based knowledge transfer across skill categories for robot control. In: Proceedings of IJCAI, pp. 3462–3468 (2017)Google Scholar
  5. 5.
    Devin, C., Gupta, A., Darrell, T., Abbeel, P., Levine, S.: Learning modular neural network policies for multi-task and multi-robot transfer. In: Proceedings IEEE ICRA (2017)Google Scholar
  6. 6.
    Fernndez, F., Garca, J., Veloso, M.: Probabilistic policy reuse for inter-task transfer learning. Rob. Auton. Syst. 58, 866–871 (2010)CrossRefGoogle Scholar
  7. 7.
    Deisenroth, M.P., Englert, P., Peters, J., Fox, D.: Multi-task policy search for robotics. In: Proceedings IEEE ICRA (2014)Google Scholar
  8. 8.
    Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E.: Adaptive mixtures of local experts. Neural Comput. 3, 79–87 (2008)CrossRefGoogle Scholar
  9. 9.
    Mülling, K., Kober, J., Kroemer, O., Peters, J.: Learning to select and generalize striking movements in robot table tennis. Int. J. Robot. Res. 32, 263–279 (2013)CrossRefGoogle Scholar
  10. 10.
    Duro, R.J., Santos, J., Becerra, J.A.: Evolving ANN controllers for smart mobile robots. In: Future Directions for Intelligent Systems and Information Sciences, pp. 34–64. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  11. 11.
    Konidaris, G., Barto, A.: Building portable options: skill transfer in reinforcement learning. In: IJCAI International Joint Conference on Artificial Intelligence, pp. 895–900 (2007)Google Scholar
  12. 12.
    Masson, W., Ranchod, P., Konidaris, G.: Reinforcement learning with parameterized actions. In: Thirtieth AAAI Conference on Artificial Intelligence (2016)Google Scholar
  13. 13.
    Da Silva, B., Konidaris, G., Barto, A.: Learning parameterized skills. In: Proceedings ICML 2012, pp. 1443–1450 (2012)Google Scholar
  14. 14.
    Bellas, F., Duro, R.J., Faiña, A., Souto, D.: Multilevel Darwinist Brain (MDB): artificial evolution in a cognitive architecture for real robots. IEEE Trans. Auton. Mental Dev. 2(4), 340–354 (2010)CrossRefGoogle Scholar
  15. 15.
    Duro, R.J., Becerra, J.A., Monroy, J., Bellas, F.: Perceptual generalization and context in a network memory inspired long term memory for artificial cognition. Int. J. Neural Syst. 29(06), 1850053 (2019)CrossRefGoogle Scholar
  16. 16.
    Prieto, A., Romero, A., Bellas, F., Salgado, R., Duro, R.J.: Introducing separable utility regions in a motivational engine for cognitive developmental robotics. Integr. Comput. Aided Eng. 26(1), 3–20 (2019)CrossRefGoogle Scholar
  17. 17.
    Harris-Warrick, R.M.: Modulation of neural networks for behaviour. Ann. Rev. Neurosci. 14, 39–57 (1991)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Alejandro Romero
    • 1
  • Francisco Bellas
    • 1
    Email author
  • Jose A. Becerra
    • 1
  • Richard J. Duro
    • 1
  1. 1.CITIC Research Center, GII, Universidade da CoruñaA CoruñaSpain

Personalised recommendations