Abstract
Action selection is extremely important, particularly when the accomplishment of competitive tasks may require access to limited motor resources. The spontaneous exploration of the world plays a fundamental role in the development of this capacity, providing subjects with an increasingly diverse set of opportunities to acquire, practice and refine the understanding of action–outcome connection. The computational modeling literature proposed a number of specific mechanisms for autonomous agents to discover and target interesting outcomes: intrinsic motivations hold a central importance among those mechanisms. Unfortunately, the study of the acquisition of action–outcome relation was mostly carried out with experiments involving extrinsic tasks, either based on rewards or on predefined task goals. This work presents a new experimental paradigm to study the effect of intrinsic motivation on action–outcome relation learning and action selection during free exploration of the world. Three- and four-year-old children were observed during the free exploration of a new toy: half of them were allowed to develop the knowledge concerning its functioning; the other half were not allowed to learn anything. The knowledge acquired during the free exploration of the toy was subsequently assessed and compared.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Approved by the local Institute Ethical Committee of the Università Campus Bio-Medico di Roma, Prot. 10.CI_REV(05).12. ComEt-CBM 07/2012.
Seq A: Lb Cb Rb Lb Cb Rb Lb Cb Rb; seq B: Cb Rb Lb Cb Rb Lb Cb Rb Lb; seq C: Rb Lb Cb Rb Lb Cb Rb Lb Cb.
References
Baldassarre G (2011) What are intrinsic motivations? A biological perspective. In: IEEE international conference on development and learning (ICDL), vol 2, pp 1–8
Baldassarre G, Mirolli M (2013) Intrinsically motivated learning in natural and artificial systems. Springer, Berlin
Baldassarre G, Mannella F, Fiore VG, Redgrave P, Gurney K, Mirolli M (2013) Intrinsically motivated action– outcome learning and goal-based action recall: a system-level bio-constrained computational model. Neural Netw 41:168–187
Balleine BW, Dickinson A (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37(4):407–419
Balleine BW, Daw ND, O'Doherty JP (2008) Multiple forms of value learning and the function of dopamine. In: Glimcher PW, Camerer CF, Poldrack RA, Fehr E (eds) Neuroeconomics: decision making and the Brain. Academic Press, New York, pp 367–385
Baranes A, Oudeyer PY (2013) Active learning of inverse models with intrinsically motivated goal exploration in robots. Robot Auton Syst 61(1):49–73
Barto A, Mirolli M, Baldassarre G (2013) Novelty or surprise? Front Psychol 4:907. doi:10.3389/fpsyg.2013.00907
Berlyne DE (1960) Conflict, arousal, and curiosity. McGraw-Hill Book, New York
Botvinick MM, Niv Y, Barto AC (2009) Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113(3):262–280
Chentanez N, Barto AG, Singh SP (2004) Intrinsically motivated reinforcement learning. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems 17. MIT Press, Cambridge, pp 1281–1288
De Charms R (1968) Personal causation: the internal affective determinants of behavior. Academic Press, New York
Di Pino G, Porcaro C, Tombini M, Assenza G, Pellegrino G, Tecchio F, Rossini P (2012) A neurally-interfaced hand prosthesis tuned inter-hemispheric communication. Restor Neurol Neurosci 30(5):407–418
Elsner B, Hommel B (2004) Contiguity and contingency in action-effect learning. Psychol Res 68(2–3):138–154
Ferreri F, Ponzo D, Vollero L, Guerra A, Di Pino G, Petrichella S, Benvenuto A, Tombini M, Rossini L, Denaro L et al (2014) Does an intraneural interface short-term implant for robotic hand control modulate sensorimotor cortical integration? An EEG-TMS co-registration study on a human amputee. Restor Neurol Neurosci 32(2):281–292
Gopnik A, Meltzoff A, Kuhl P (1999) The scientist in the crib: what early learning tells us about the mind. Harper-Collins, New York, p 1
Gopnik A, Glymour C, Sobel DM, Schulz LE, Kushnir T, Danks D (2004) A theory of causal learning in children: causal maps and bayes nets. Psychol Rev 111(1):3
Gottlieb J, Oudeyer PY, Lopes M, Baranes A (2013) Information-seeking, curiosity, and attention: computational and neural mechanisms. Trends Cogn Sci 17(11):585–593
Hickling AK, Wellman HM (2001) The emergence of children’s causal explanations and theories: evidence from everyday conversation. Dev Psychol 37(5):668
Oudeyer PY, Kaplan F (2007) What is intrinsic motivation? a typology of computational approaches. Front Neurosci 1:6
Keen R (2011) The development of problem solving in young children: a critical cognitive skill. Annu Rev Psychol 62:1–21
Kenward B, Folke S, Holmberg J, Johansson A, Gredebäck G (2009) Goal directedness and decision making in infants. Dev Psychol 45(3):809
Kumaran D, Maguire EA (2007) Which computational mechanisms operate in the hippocampus during novelty detection? Hippocampus 17(9):735–748
Mascalzoni E, Regolin L, Vallortigara G, Simion F (2013) The cradle of causal reasoning: newborns preference for physical causality. Dev Sci 16(3):327–335
Mirolli M, Baldassarre G (2013) Functions and mechanisms of intrinsic motivations: the knowledge versus competence distinction. In: Baldassarre G, Mirolli M (eds) Intrinsically motivated learning in natural and artificial systems. Springer, Berlin, pp 49-72
Mirolli M, Santucci V, Baldassarre G (2013) Phasic dopamine as a prediction error signal of intrinsic and extrinsic reinforcements: a computational model. Neural Netw 39:40–51
Moll H, Tomasello M (2010) Infant cognition. Curr Biol 20(20):R872–R875
Nehmzow U, Gatsoulis Y, Kerr E, Condell J, Siddique N, McGinnity TM (2013) Novelty detection as an intrinsic motivation for cumulative learning robots. In: Baldassarre G, Mirolli M (eds) Intrinsically motivated learning in natural and artificial systems. Springer, New York, pp 185–207
Piaget J, Cook MT (1952) The origins of intelligence in children. W.W. Norton, New York
Polich J (2007) Updating p300: an integrative theory of p3a and p3b. Clin Neurophysiol 118(10):2128–2148
Ranganath C, Rainer G (2003) Neural mechanisms for detecting and remembering novel events. Nat Rev Neurosci 4(3):193–202
Redgrave P, Gurney K (2006) The short-latency dopamine signal: a role in discovering novel actions? Nat Rev Neurosci 7(12):967–975
Redgrave P, Vautrelle N, Reynolds J (2011) Functional properties of the basal ganglia’s re-entrant loop architecture: selection and reinforcement. Neuroscience 198:138–151
Ryan RM, Deci EL (2000) Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp Educ Psychol 25(1):54–67
Santucci VG, Baldassarre G, Mirolli M (2013) Which is the best intrinsic motivation signal for learning multiple skills? Front Neurorobot 7:22. doi:10.3389/fnbot.2013.00022
Schembri M, Mirolli M, Baldassarre G (2007) Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. In: IEEE 6th international conference on development and learning (ICDL 2007), pp 282–287
Schmidhuber J (1991) Curious model-building control systems. In: IEEE international joint conference on neural networks, pp 1458–1463
Singh S, Lewis RL, Barto AG, Sorg J (2010) Intrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Trans Auton Ment Dev 2(2):70–82
Smith L, Gasser M (2005) The development of embodied cognition: six lessons from babies. Artif Life 11(1–2):13–29
Spelke ES, Breinlinger K, Macomber J, Jacobson K (1992) Origins of knowledge. Psychol Rev 99(4):605
Taffoni F, Formica D, Zompanti A, Mirolli M, Balsassarre G, Keller F, Guglielmelli E (2012a) A mechatronic platform for behavioral studies on infants. In: 4th IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics (BioRob), pp 1874–1878
Taffoni F, Vespignani M, Formica D, Cavallo G, Di Sorrentino EP, Sabbatini G, Truppa V, Mirolli M, Baldassarre G, Visalberghi E et al (2012b) A mechatronic platform for behavioral analysis on nonhuman primates. J Integr Neurosci 11(01):87–101
Taffoni F, Formica D, Schiavone G, Scorcia M, Tomassetti A, di Sorrentino EP, Sabbatini G, Truppa V, Mannella F, Fiore V et al (2013) The “mechatronic board”: a tool to study intrinsic motivations in humans, monkeys, and humanoid robots. In: Baldassarre G, Mirolli M (eds) Intrinsically motivated learning in natural and artificial systems. Springer, New York, pp 411–432
Téglás E, Vul E, Girotto V, Gonzalez M, Tenenbaum JB, Bonatti LL (2011) Pure reasoning in 12-month-old infants as probabilistic inference. Science 332(6033):1054–1059
von Hofsten C (2004) An action perspective on motor development. Trends Cogn Sci 8(6):266–272
White RW (1959) Motivation reconsidered: the concept of competence. Psychol Rev 66(5):297
Acknowledgments
This work was funded by FP7-ICT program (Project No. ICT-2007.3.2-231722-IM-CLeVeR). The authors would like to thank Dr. Roberta Aronica for proof-reading the document, and the anonymous reviewers for their valuable comments and suggestions to improve the work.
Author information
Authors and Affiliations
Corresponding author
Additional information
Fabrizio Taffoni and Eleonora Tamilia have equally contributed to this work.
Rights and permissions
About this article
Cite this article
Taffoni, F., Tamilia, E., Focaroli, V. et al. Development of goal-directed action selection guided by intrinsic motivations: an experiment with children. Exp Brain Res 232, 2167–2177 (2014). https://doi.org/10.1007/s00221-014-3907-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-014-3907-z