Cognitive Neurodynamics

, Volume 6, Issue 1, pp 43–59 | Cite as

Neuro-robotics study on integrative learning of proactive visual attention and motor behaviors

  • Sungmoon Jeong
  • Hiroaki Arie
  • Minho Lee
  • Jun Tani
Research Article


The current paper proposes a novel model for integrative learning of proactive visual attention and sensory-motor control as inspired by the premotor theory of visual attention. The model is characterized by coupling a slow dynamics network with a fast dynamics network and by inheriting our prior proposed multiple timescales recurrent neural networks model (MTRNN) that may correspond to the fronto-parietal networks in the cortical brains. The neuro-robotics experiments in a task of manipulating multiple objects utilizing the proposed model demonstrated that some degrees of generalization in terms of position and object size variation can be achieved by organizing seamless integration of the proactive object-related visual attention and the related sensory-motor control into a set of action primitives in the distributed neural activities appearing in the fast dynamics network. It was also shown that such action primitives can be combined in compositional ways in acquiring novel actions in the slow dynamics network. The experimental results presented substantiate the premotor theory of visual attention.


Proactive visual attention Multiple time recurrent neural networks Premotor theory of visual attention Multiple objects manipulation task 



This research was supported by the Ministry of Education, Culture, Sports, Science and Technology of Japan, Grants-in-Aid for Scientific Research on Innovative Areas (No. 22120523, 2010) (50%), the Korea-Japan Collaboration Core Project on Neuroinformatics (10%) and the Converging Research Center Program (40%) through the National Research Foundation (NRF) of Korea funded by the Ministry of Education, Science and Technology (2009-0082262).


  1. Ballard DH (1991) Animate vision. Artif Intell J 48:57–86CrossRefGoogle Scholar
  2. Ballard DH, Hayhoe MM, Li F, Whitehead SD, Frisby JP, Taylor JG, Fisher RB (1992) Hand-eye coordination during sequential tasks [and discussion]. Philos Trans Biol Sci R Soc 337:331–339CrossRefGoogle Scholar
  3. Ballard DH, Hayhoe MM, Pelz JB (1995) Memory representations in natural tasks. J Cogn Neurosci 7:66–80CrossRefGoogle Scholar
  4. Beer RD (2000) Dynamical approaches to cognitive science. Trends Cogn Sci 4(3):91–99PubMedCrossRefGoogle Scholar
  5. Berkinblit MB, Fookson OI, Smetanin B, Adamovich SV, Poizner H (1995) The interaction of visual and proprioceptive inputs in pointing to actual and remembered targets. Exp Brain Res 107:326–330PubMedCrossRefGoogle Scholar
  6. Bowman MC, Johannson RS (2009) Eye-hand coordination in a sequential target contract task. Exp Brain Res 195:273–283PubMedCrossRefGoogle Scholar
  7. Carlton LG (1981) Processing visual feedback information for movement control. J Exp Psychol Hum Percept Perform 7:1019–1030PubMedCrossRefGoogle Scholar
  8. Carpenter GA, Grossberg S, Markuzon N, Reynolds JH, Rosen DB (1992) Fuzzy ARTMAP: a neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Trans Neural Netw 3(5):698–713PubMedCrossRefGoogle Scholar
  9. Choi SB, Jung BS, Ban SW, Niitsuma H, Lee M (2006) Biologically motivated vergence control system using human-like selective attention model. Neurocomputing 69:537–558CrossRefGoogle Scholar
  10. Connolly JD, Goodale MA, Menon RS, Munoz DP (2002) Human fMRI evidence for the neural correlates of preparatory set. Nat Neurosci 5:1345–1352PubMedCrossRefGoogle Scholar
  11. Corbetta M (1998) Frontoparietal cortical networks for directing attention and the eye to visual locations: Identical, independent, or overlapping neural systems? Proc Natl Acad Sci 95:831–838PubMedCrossRefGoogle Scholar
  12. Corbetta M, Shulman GL (2002) Control of goal-directed and stimulus-driven attention in the brain. Nat Rev Neurosci 3:201–215PubMedCrossRefGoogle Scholar
  13. Corbetta M, Akbudak E, Conturo TE, Snyder AZ, Ollinger JM, Drury HA, Linenweber MR, Petersen SE, Raichle ME, Van Essen DC, Shulman GL (1998) A common network of functional areas for attention and eye movements. Neuron 21:761–773PubMedCrossRefGoogle Scholar
  14. Craighero L, Fadiga L, Rizzolatti G, Umiltà C (1999) Action for perception: a motor-visual attentional effect. J Exp Psychol Hum Percept Perform 25:1673–1692PubMedCrossRefGoogle Scholar
  15. Crawford JD, Medendorp WP, Marotta JJ (2004) Spatial transformations for eye-hand coordination. J Neurophysiol 92:10–19PubMedCrossRefGoogle Scholar
  16. Doya K, Yoshizawa S (1989) Memorizing oscillatory patterns in the analog neuron network. In: Proceedings of 1989 international joint conference on neural networks, vol I, Washington, DC, pp 27–32Google Scholar
  17. Ehrsson H, Fagergren A, Johansson R, Forssberg H (2003) Evidence for the involvement of the posterior parietal cortex in coordination of fingertip forces for grasp stability in manipulation. J Neurophysiol 90:2978–2986PubMedCrossRefGoogle Scholar
  18. Eskandar E, Assad J (1999) Dissociation of visual, motor and predictive signals in parietal cortex during visual guidance. Nat Neurosci 2:88–93PubMedCrossRefGoogle Scholar
  19. Friston K, Mattout J, Kilner J (2011) Action understanding and active inference. Biol Cybern 104:137–160PubMedCrossRefGoogle Scholar
  20. Furneaux S, Land MF (1999) The effects of skill on the eye-hand span during musical sight-reading. In: Proceeding of Royal Society, vol 266, pp 2435–2440Google Scholar
  21. Fuster JM (2008) The prefrontal cortex, 4th edn. Academic Press, LondonGoogle Scholar
  22. Hayhoe MM, Shrivastava A, Mruczek R, Pelz JB (2003) Visual memory and motor planning in a natural task. J Vis 3:49–63PubMedCrossRefGoogle Scholar
  23. Herst AN, Epelboim J, Steinman RM (2001) Temporal coordination of the human head and eye during a natural sequential tapping task. Vis Res 41:3307–3319PubMedCrossRefGoogle Scholar
  24. Jellema T, Baker CI, Wicker B, Perrett DI (2000) Neural representation for the perception of the intentionality of actions. Brain Cogn 44:280–302PubMedCrossRefGoogle Scholar
  25. Jeong S, Ban S-W, Lee M (2008) Stereo saliency map considering affective factors and selective motion analysis in a dynamic environment. Neural Netw 21:1420–1430PubMedCrossRefGoogle Scholar
  26. Johansson RS, Westling G, Ba¨ ckstro¨m A, Flanagan JR (2001) Eye-hand coordination in object manipulation. J Neurosci 21:6917–6932PubMedGoogle Scholar
  27. Kohonen T (2001) Self-organizing maps. Springer Series in Information Sciences, vol 30, SpringerGoogle Scholar
  28. Kustov AA, Robinson DL (1996) Shared neural control of attentional shifts and eye movements. Nature 384:74–77PubMedCrossRefGoogle Scholar
  29. Land MF, McLeod P (2000) From eye movements to actions: how batsmen hit the ball. Nat Neurosci 3:1340–1345PubMedCrossRefGoogle Scholar
  30. Land M, Mennie N, Rusted J (1999) The roles of vision and eye movements in the control of activities of daily living. Perception 28:1311–1328PubMedCrossRefGoogle Scholar
  31. Lennie P (2003) The cost of cortical computation. Curr Biol 13:493–497PubMedCrossRefGoogle Scholar
  32. Linda JL, Susan LD (2009) Modelling attention in individual cells leads to a system with realistic saccade behaviours. Cogn Neurodyn 3:223–242CrossRefGoogle Scholar
  33. McCallum AK (1996) Learning to use selective attention and short-term memory in sequential tasks. In: Proceedings of the fourth international conference on simulation of adaptive behavior, (SAB’96), pp 315–324Google Scholar
  34. Metta G, Sandini G, Konczak J (1999) A developmental approach to visually-guided reaching in artificial systems. Neural Netw 12(10):1413–1427PubMedCrossRefGoogle Scholar
  35. Moore T, Fallah M (2001) Control of eye movements and spatial attention. Proc Natl Acad Sci 98:1273–1276PubMedCrossRefGoogle Scholar
  36. Mulliken GH, Musallam S, Andersen RA (2008) Forward estimation of movement state in posterior parietal cortex. Proc Natl Acad Sci USA 105(24):8170–8177PubMedCrossRefGoogle Scholar
  37. Nishimoto R, Tani J (2009) Development of hierarchical structures for actions and motor imagery: a constructivist view from synthetic neurorobotics study. Psychol Res 73:545–558PubMedCrossRefGoogle Scholar
  38. Nobre AC, Gitelman DR, Dias EC, Mesulam MM (2000) Covert visual spatial orienting and saccades: overlapping neural systems. Neuroimage 11:210–216PubMedCrossRefGoogle Scholar
  39. Paillard J (1996) Fast and slow feedback loops for the visual correction of spatial errors in a pointing task: a reappraisal. Can J Physiol Pharmacol 74:401–417PubMedCrossRefGoogle Scholar
  40. Pelz J, Hayhoe M, Loeber R (2001) The coordination of eye, head, and hand movements in a natural task. Exp Brain Res 139:266–277PubMedCrossRefGoogle Scholar
  41. Posner MI, Dehaene S (1994) Attentional networks. Trends Neurosci 17:75–79PubMedCrossRefGoogle Scholar
  42. Prablanc C, Martin O (1992) Automatic control during hand reaching at undetected two-dimensional target displacements. J Neurophysiol 67:455–469PubMedGoogle Scholar
  43. Prablanc C, Echallier JF, Komilis E, Jeannerod M (1979) Optimal response of eye and hand motor systems in pointing at a visual target. I. Spatio-temporal characteristics of eye and hand movements and their relationships when varying the amount of visual information. Biol Cybern 35:113–124PubMedCrossRefGoogle Scholar
  44. Prablanc C, Pelisson D, Goodale MA (1986) Visual control of reaching movements without vision of the limb. 1. Role of retinal feedback of target position in guiding the hand. Exp Brain Res 62:293–302PubMedCrossRefGoogle Scholar
  45. Prablanc C, Desmurget M, Grea H (2003) Neural control of on-line guidance of hand-reaching movements. Prog Brain Res 142:155–170PubMedCrossRefGoogle Scholar
  46. Rao RPN (1995) An active vision architecture based on iconic representations. Artif Intell 78:461–505CrossRefGoogle Scholar
  47. Rao RPN, Ballard DH (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2:79–87PubMedCrossRefGoogle Scholar
  48. Rizzolatti G, Craighero L (1998) Spatial attention: mechanisms and theories. In: Sabourin M, Craik F, Robert M (eds) Advances in psychological science, 2, biological and cognitive aspects. Psychology Press, East Sussex, pp 171–198Google Scholar
  49. Rizzolatti G, Riggio L, Dascola I, Umiltá C (1987) Reorienting attention across the horizontal and vertical meridians: evidence in favor of a premotor theory of attention. Neuropsychologia 25:31–40PubMedCrossRefGoogle Scholar
  50. Rizzolatti G, Riggio L, Sheliga BM (1994) Space and selective attention. In: Umiltà C, Moscovitch M (eds) Attention and performance XV. MIT Press, Cambridge, pp 231–265Google Scholar
  51. Rizzolatti G, Fadiga L, Galless V, Fogassi L (1996) Premotor cortex and the recognition of motor actions. Cogn Brain Res 3:131–141CrossRefGoogle Scholar
  52. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. Parallel distributed processing, explorations in the microstructure of cognition, 1st edn. MIT Press, CambridgeGoogle Scholar
  53. Saarinen J, Kohonen T (1985) Self-organized formation of colour maps in a model cortex. Perception 14(6):711–719PubMedCrossRefGoogle Scholar
  54. Sarlegna F, Blouin J, Vercher JL, Bresciani JP, Bourdin C, Gauthier GM (2004) Online control of the direction of rapid reaching movements. Exp Brain Res 157:468–471PubMedCrossRefGoogle Scholar
  55. Saunders JA, Knill DC (2004) Visual feedback control of hand movements. J Neurosci 24:3223–3234PubMedCrossRefGoogle Scholar
  56. Schubotz RI, von Cramon DY (2001) Functional organization of the lateral premotor cortex: fMRI reveals different regions activated by anticipation of object properties, location and speed. Cogn Brain Res 11(1):97–112CrossRefGoogle Scholar
  57. Seltzer B, Pandya DN (1994) Parietal, temporal, and occipita projections to cortex of the superior temporal sulcus in the rhesus monkey: a retrograde tracer study. J Comp Neurol 343(3):445–463PubMedCrossRefGoogle Scholar
  58. Sheliga BM, Riggio L, Rizzolatti G (1995) Spatial attention and eye movements. Exp Brain Res 105:261–275PubMedCrossRefGoogle Scholar
  59. Smeets JB, Hayhoe MM, Ballard DH (1996) Goal-directed arm movements change eye-head coordination. Exp Brain Res 109:434–440PubMedCrossRefGoogle Scholar
  60. Suzuki M, Floreano D (2008) Enactive robot vision. Adaptive Behavior 16(2–3):122–128CrossRefGoogle Scholar
  61. Tanji J, Shima K (1994) Role for supplementary motor area cells in planning several movements ahead. Nature 371:413–416PubMedCrossRefGoogle Scholar
  62. Tsotsos JK (1990) Analyzing vision at the complexity level. Behav Brain Sci 13:423–469CrossRefGoogle Scholar
  63. Tsubomi H, Ikeda T, Hanakawa T, Hirose N, Fukuyama H, Osaka N (2009) Connectivity and signal intensity in the parieto-occipital cortex predicts top–down attentional effect in visual masking: An fMRI study based on individual differences. NeuroImage 45:587–597PubMedCrossRefGoogle Scholar
  64. Varela FJ, Thompson E, Rosch E (1991) The embodied mind. MIT Press, CambridgeGoogle Scholar
  65. Williams RJ, Zipser D (1989) A learning algorithm for continually running fully recurrent neural networks. Neural Comput 1(2):270–280CrossRefGoogle Scholar
  66. Wolpert D, Kawato M (1998) Multiple paired forward and inverse models for motor control. Neural Netw 11:1317–1329PubMedCrossRefGoogle Scholar
  67. Yamashita Y, Tani J (2008) Emergence of functional hierarchy in a multiple timescale neural network model: A humanoid robot experiment. PLoS Comput Biol 4(11):1–18CrossRefGoogle Scholar
  68. Yuqiao G, Hans L (2007) A neural network model of attention-modulated neurodynamics. Cogn Neurodyn 1:275–285CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2011

Authors and Affiliations

  • Sungmoon Jeong
    • 1
  • Hiroaki Arie
    • 2
  • Minho Lee
    • 1
  • Jun Tani
    • 2
  1. 1.School of Electronics EngineeringKyungpook National UniversityTaeguKorea
  2. 2.Laboratory for Behavior and Dynamic CognitionRIKEN Brain Science InstituteWako-shiJapan

Personalised recommendations