Advertisement

Experimental Brain Research

, Volume 232, Issue 7, pp 2167–2177 | Cite as

Development of goal-directed action selection guided by intrinsic motivations: an experiment with children

  • Fabrizio TaffoniEmail author
  • Eleonora Tamilia
  • Valentina Focaroli
  • Domenico Formica
  • Luca Ricci
  • Giovanni Di Pino
  • Gianluca Baldassarre
  • Marco Mirolli
  • Eugenio Guglielmelli
  • Flavio Keller
Research Article

Abstract

Action selection is extremely important, particularly when the accomplishment of competitive tasks may require access to limited motor resources. The spontaneous exploration of the world plays a fundamental role in the development of this capacity, providing subjects with an increasingly diverse set of opportunities to acquire, practice and refine the understanding of action–outcome connection. The computational modeling literature proposed a number of specific mechanisms for autonomous agents to discover and target interesting outcomes: intrinsic motivations hold a central importance among those mechanisms. Unfortunately, the study of the acquisition of action–outcome relation was mostly carried out with experiments involving extrinsic tasks, either based on rewards or on predefined task goals. This work presents a new experimental paradigm to study the effect of intrinsic motivation on action–outcome relation learning and action selection during free exploration of the world. Three- and four-year-old children were observed during the free exploration of a new toy: half of them were allowed to develop the knowledge concerning its functioning; the other half were not allowed to learn anything. The knowledge acquired during the free exploration of the toy was subsequently assessed and compared.

Keywords

Intrinsic motivation Action selection Curiosity  Action–outcome contingency Novelty detection 

Notes

Acknowledgments

This work was funded by FP7-ICT program (Project No. ICT-2007.3.2-231722-IM-CLeVeR). The authors would like to thank Dr. Roberta Aronica for proof-reading the document, and the anonymous reviewers for their valuable comments and suggestions to improve the work.

References

  1. Baldassarre G (2011) What are intrinsic motivations? A biological perspective. In: IEEE international conference on development and learning (ICDL), vol 2, pp 1–8Google Scholar
  2. Baldassarre G, Mirolli M (2013) Intrinsically motivated learning in natural and artificial systems. Springer, BerlinCrossRefGoogle Scholar
  3. Baldassarre G, Mannella F, Fiore VG, Redgrave P, Gurney K, Mirolli M (2013) Intrinsically motivated action– outcome learning and goal-based action recall: a system-level bio-constrained computational model. Neural Netw 41:168–187Google Scholar
  4. Balleine BW, Dickinson A (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37(4):407–419PubMedCrossRefGoogle Scholar
  5. Balleine BW, Daw ND, O'Doherty JP (2008) Multiple forms of value learning and the function of dopamine. In: Glimcher PW, Camerer CF, Poldrack RA, Fehr E (eds) Neuroeconomics: decision making and the Brain. Academic Press, New York, pp 367–385Google Scholar
  6. Baranes A, Oudeyer PY (2013) Active learning of inverse models with intrinsically motivated goal exploration in robots. Robot Auton Syst 61(1):49–73CrossRefGoogle Scholar
  7. Barto A, Mirolli M, Baldassarre G (2013) Novelty or surprise? Front Psychol 4:907. doi: 10.3389/fpsyg.2013.00907
  8. Berlyne DE (1960) Conflict, arousal, and curiosity. McGraw-Hill Book, New YorkCrossRefGoogle Scholar
  9. Botvinick MM, Niv Y, Barto AC (2009) Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113(3):262–280PubMedCentralPubMedCrossRefGoogle Scholar
  10. Chentanez N, Barto AG, Singh SP (2004) Intrinsically motivated reinforcement learning. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems 17. MIT Press, Cambridge, pp 1281–1288Google Scholar
  11. De Charms R (1968) Personal causation: the internal affective determinants of behavior. Academic Press, New YorkGoogle Scholar
  12. Di Pino G, Porcaro C, Tombini M, Assenza G, Pellegrino G, Tecchio F, Rossini P (2012) A neurally-interfaced hand prosthesis tuned inter-hemispheric communication. Restor Neurol Neurosci 30(5):407–418PubMedGoogle Scholar
  13. Elsner B, Hommel B (2004) Contiguity and contingency in action-effect learning. Psychol Res 68(2–3):138–154PubMedCrossRefGoogle Scholar
  14. Ferreri F, Ponzo D, Vollero L, Guerra A, Di Pino G, Petrichella S, Benvenuto A, Tombini M, Rossini L, Denaro L et al (2014) Does an intraneural interface short-term implant for robotic hand control modulate sensorimotor cortical integration? An EEG-TMS co-registration study on a human amputee. Restor Neurol Neurosci 32(2):281–292Google Scholar
  15. Gopnik A, Meltzoff A, Kuhl P (1999) The scientist in the crib: what early learning tells us about the mind. Harper-Collins, New York, p 1Google Scholar
  16. Gopnik A, Glymour C, Sobel DM, Schulz LE, Kushnir T, Danks D (2004) A theory of causal learning in children: causal maps and bayes nets. Psychol Rev 111(1):3PubMedCrossRefGoogle Scholar
  17. Gottlieb J, Oudeyer PY, Lopes M, Baranes A (2013) Information-seeking, curiosity, and attention: computational and neural mechanisms. Trends Cogn Sci 17(11):585–593Google Scholar
  18. Hickling AK, Wellman HM (2001) The emergence of children’s causal explanations and theories: evidence from everyday conversation. Dev Psychol 37(5):668PubMedCrossRefGoogle Scholar
  19. Oudeyer PY, Kaplan F (2007) What is intrinsic motivation? a typology of computational approaches. Front Neurosci 1:6Google Scholar
  20. Keen R (2011) The development of problem solving in young children: a critical cognitive skill. Annu Rev Psychol 62:1–21PubMedCrossRefGoogle Scholar
  21. Kenward B, Folke S, Holmberg J, Johansson A, Gredebäck G (2009) Goal directedness and decision making in infants. Dev Psychol 45(3):809PubMedCrossRefGoogle Scholar
  22. Kumaran D, Maguire EA (2007) Which computational mechanisms operate in the hippocampus during novelty detection? Hippocampus 17(9):735–748PubMedCrossRefGoogle Scholar
  23. Mascalzoni E, Regolin L, Vallortigara G, Simion F (2013) The cradle of causal reasoning: newborns preference for physical causality. Dev Sci 16(3):327–335Google Scholar
  24. Mirolli M, Baldassarre G (2013) Functions and mechanisms of intrinsic motivations: the knowledge versus competence distinction. In: Baldassarre G, Mirolli M (eds) Intrinsically motivated learning in natural and artificial systems. Springer, Berlin, pp 49-72Google Scholar
  25. Mirolli M, Santucci V, Baldassarre G (2013) Phasic dopamine as a prediction error signal of intrinsic and extrinsic reinforcements: a computational model. Neural Netw 39:40–51Google Scholar
  26. Moll H, Tomasello M (2010) Infant cognition. Curr Biol 20(20):R872–R875PubMedCrossRefGoogle Scholar
  27. Nehmzow U, Gatsoulis Y, Kerr E, Condell J, Siddique N, McGinnity TM (2013) Novelty detection as an intrinsic motivation for cumulative learning robots. In: Baldassarre G, Mirolli M (eds) Intrinsically motivated learning in natural and artificial systems. Springer, New York, pp 185–207Google Scholar
  28. Piaget J, Cook MT (1952) The origins of intelligence in children. W.W. Norton, New YorkCrossRefGoogle Scholar
  29. Polich J (2007) Updating p300: an integrative theory of p3a and p3b. Clin Neurophysiol 118(10):2128–2148PubMedCentralPubMedCrossRefGoogle Scholar
  30. Ranganath C, Rainer G (2003) Neural mechanisms for detecting and remembering novel events. Nat Rev Neurosci 4(3):193–202PubMedCrossRefGoogle Scholar
  31. Redgrave P, Gurney K (2006) The short-latency dopamine signal: a role in discovering novel actions? Nat Rev Neurosci 7(12):967–975PubMedCrossRefGoogle Scholar
  32. Redgrave P, Vautrelle N, Reynolds J (2011) Functional properties of the basal ganglia’s re-entrant loop architecture: selection and reinforcement. Neuroscience 198:138–151PubMedCrossRefGoogle Scholar
  33. Ryan RM, Deci EL (2000) Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp Educ Psychol 25(1):54–67PubMedCrossRefGoogle Scholar
  34. Santucci VG, Baldassarre G, Mirolli M (2013) Which is the best intrinsic motivation signal for learning multiple skills? Front Neurorobot 7:22. doi: 10.3389/fnbot.2013.00022
  35. Schembri M, Mirolli M, Baldassarre G (2007) Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. In: IEEE 6th international conference on development and learning (ICDL 2007), pp 282–287Google Scholar
  36. Schmidhuber J (1991) Curious model-building control systems. In: IEEE international joint conference on neural networks, pp 1458–1463Google Scholar
  37. Singh S, Lewis RL, Barto AG, Sorg J (2010) Intrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Trans Auton Ment Dev 2(2):70–82CrossRefGoogle Scholar
  38. Smith L, Gasser M (2005) The development of embodied cognition: six lessons from babies. Artif Life 11(1–2):13–29PubMedCrossRefGoogle Scholar
  39. Spelke ES, Breinlinger K, Macomber J, Jacobson K (1992) Origins of knowledge. Psychol Rev 99(4):605PubMedCrossRefGoogle Scholar
  40. Taffoni F, Formica D, Zompanti A, Mirolli M, Balsassarre G, Keller F, Guglielmelli E (2012a) A mechatronic platform for behavioral studies on infants. In: 4th IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics (BioRob), pp 1874–1878Google Scholar
  41. Taffoni F, Vespignani M, Formica D, Cavallo G, Di Sorrentino EP, Sabbatini G, Truppa V, Mirolli M, Baldassarre G, Visalberghi E et al (2012b) A mechatronic platform for behavioral analysis on nonhuman primates. J Integr Neurosci 11(01):87–101PubMedCrossRefGoogle Scholar
  42. Taffoni F, Formica D, Schiavone G, Scorcia M, Tomassetti A, di Sorrentino EP, Sabbatini G, Truppa V, Mannella F, Fiore V et al (2013) The “mechatronic board”: a tool to study intrinsic motivations in humans, monkeys, and humanoid robots. In: Baldassarre G, Mirolli M (eds) Intrinsically motivated learning in natural and artificial systems. Springer, New York, pp 411–432Google Scholar
  43. Téglás E, Vul E, Girotto V, Gonzalez M, Tenenbaum JB, Bonatti LL (2011) Pure reasoning in 12-month-old infants as probabilistic inference. Science 332(6033):1054–1059PubMedCrossRefGoogle Scholar
  44. von Hofsten C (2004) An action perspective on motor development. Trends Cogn Sci 8(6):266–272PubMedCrossRefGoogle Scholar
  45. White RW (1959) Motivation reconsidered: the concept of competence. Psychol Rev 66(5):297PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Fabrizio Taffoni
    • 1
    Email author
  • Eleonora Tamilia
    • 1
  • Valentina Focaroli
    • 2
  • Domenico Formica
    • 1
  • Luca Ricci
    • 1
  • Giovanni Di Pino
    • 3
  • Gianluca Baldassarre
    • 4
  • Marco Mirolli
    • 4
  • Eugenio Guglielmelli
    • 1
  • Flavio Keller
    • 2
  1. 1.Laboratory of Biomedical Robotics and BiomicrosystemsUniversità Campus Bio-Medico di RomaRomeItaly
  2. 2.Laboratory of Developmental Neuroscience and Neural PlasticityUniversità Campus Bio-Medico di RomaRomeItaly
  3. 3.Laboratory of Biomedical Robotics and Biomicrosystems, Institute of Neurology, Fondazione Alberto Sordi - Research Institute for AgeingUniversità Campus Bio-Medico di RomaRomeItaly
  4. 4.Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and TechnologiesCNRRomeItaly

Personalised recommendations