Biological Cybernetics

, Volume 73, Issue 3, pp 265–274 | Cite as

Complex sensory-motor sequence learning based on recurrent state representation and reinforcement learning

  • Peter F. Dominey
Original Papers

Abstract

A novel neural network model is presented that learns by trial-and-error to reproduce complex sensory-motor sequences. One subnetwork, corresponding to the prefrontal cortex (PFC), is responsible for generating unique patterns of activity that represent the continuous state of sequence execution. A second subnetwork, corresponding to the striatum, associates these state-encoding patterns with the correct response at each point in the sequence execution. From a neuroscience perspective, the model is based on the known cortical and subcortical anatomy of the primate oculomotor system. From a theoretical perspective, the architecture is similar to that of a finite automaton in which outputs and state transitions are generated as a function of inputs and the current state. Simulation results for complex sequence reproduction and sequence discrimination are presented.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alexander GE, DeLong MR, Strick PL (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci 9:357–381CrossRefPubMedGoogle Scholar
  2. Andersen RA, Asanuma C, Cowan MW (1985) Callosal and prefrontal associational projecting cell populations in area 7a of the macaque monkey: a study using retrogradely transported flourescent dyes. J Comp Neurol 232:443–455CrossRefPubMedGoogle Scholar
  3. Barash S, Bracewell RM, Fogassi L, Gnadt JW, Andersen RA (1991) Saccade-related activity in the lateral intraparietal area II. Spatial properties. J Neurophysiol 66:1109–1124PubMedGoogle Scholar
  4. Barone P, Joseph J-P (1989) Prefrontal cortex and spatial sequencing in macaque moneky. Exp Brain Res 78:447–464CrossRefPubMedGoogle Scholar
  5. Barto AG (1990) Connectionist learning for control: an overview. In: Miller WT, Sutton RS, Werbos PJ (eds) Neural Networks for Control. MIT Press, Cambridge MA, pp 1–58Google Scholar
  6. Calabresi P, Maj R, Pisani A, Mercuir NB, Bernardi G (1992) Long-term synaptic depression in the striatum: physiological and pharmacological characterization. J Neurosci 12:4224–4233PubMedGoogle Scholar
  7. Carlsson M, Carlsson A (1990) Interactions between glutamatergic and monoaminergic systems within the basal ganglia — indications for schizophrenia and Parkinson's disease. Trends Neurosci 13:272–276CrossRefPubMedGoogle Scholar
  8. Chevalier G, Vacher S, Deniau JM, Desban M (1985) Disinhibition as a basic process in the expression of striatal functions. I. The striato-nigral influence on the tecto-spinal tecto-diencephalic neurons. Brain Res 334:215–226CrossRefPubMedGoogle Scholar
  9. Dehaene S, Changeux J-P, Nadal J-P (1987) Neural networks that learn temporal sequences by preselection. Proc Natl Acad Sci 84:2727–2731PubMedGoogle Scholar
  10. Dominey PF (1993) Models of spatial accuracy and conditional behavior in oculomotor control. PhD Thesis, Dept of Computer Science, University of Southern CaliforniaGoogle Scholar
  11. Dominey PF, Arbib MA (1992) A cortico-subcortical model for generation of spatially accurate sequential saccades. Cerebral Cortex 2:153–175PubMedGoogle Scholar
  12. Dominey PF, Arbib MA, Joseph JP (1995) A model of corticostriatal plasticity for learning oculomotor associations and sequences. J Cog Neuroscience 7:311–336Google Scholar
  13. Goldman-Rakic PS (1987) Circuitry of primate prefrontal cortex and regulation of behavior by representational memory. In: Vernon B Mountcastle (eds) Handbook of physiology. The Nervous system. Vol. V, Chap 9. 9: 373–417Google Scholar
  14. Herz A, Sulzer B, Kühn R, van Hemmen JL (1989) Hebbian learning reconsidered: representation of static and dynamic objects in associative neural nets. Biol Cybern 60:457–467CrossRefPubMedGoogle Scholar
  15. Hikosaka O, Sakamoto M, Usui S (1989) Functional properties of monkey caudate neurons. I. Activities related to saccadic eye movements. J Neurophysiol 61:780–798PubMedGoogle Scholar
  16. Keele SW, Jennings P, Jones S, Caulton D, Cohen A (1995) On the modularity of sequence representation. J Motor Behavior 27:17–30Google Scholar
  17. Kermadi I, Jurquet Y, Arzi M, Joseph J-P (1993) Neural activity in the caudate nucleus of monkeys during spatial sequencing. Exp Brain Res 94:352–356CrossRefPubMedGoogle Scholar
  18. Kühn R, van Hemmen JL (1992) Temporal association. In: Domanay E, Hemmen van JL, Schulten K (eds) Physics of neural networks. Springer, Berlin Heidelberg New York, pp 213–280Google Scholar
  19. Ljungberg T, Apicella P, Schultz W (1991) Responses of monkey midbrain dopamine neurons during delayed alternation performance. Brain Res 567:337–341CrossRefPubMedGoogle Scholar
  20. Mercuri M, Bernardi G, Calabresi P, Cotugno A, Levi G, Stanzione P (1985) Dopamine decreases cell excitability in rat striatal neurons by pre- and postsynaptic mechanisms. Brain Res 385: 110–121CrossRefGoogle Scholar
  21. Munoz DP, Wurtz RH (1992) Role of the rostral superior colliculus in active visual fixation on execution of express saccades. J Neurophysiol 67:1000–1002PubMedGoogle Scholar
  22. Reading PJ, Dunnet SB, Robins TW (1991) Dissociable roles of the ventral, medial and lateral striatum on the acquisition and performance of a complex visual stimulus-response habit. Behav Brain Res 45:147–161PubMedGoogle Scholar
  23. Robins TW, Giardini V, Jones GH, Reading PJ, Sahakian BJ (1990) Effects of dopamine depletion from the caudate-putamen and nucleus accumbes septi on the acquisition and performance of a conditional discrimination task. Behav Brain Res 38:243–261CrossRefPubMedGoogle Scholar
  24. Rolls ET, Williams GV (1987) Sensory and movement-related neuronal activity in different regions of the primate striatum. In: Schneider JS, Lidsky TI (eds) Basal ganglia and motor behavior. Hans Huner, Toronto, pp 37–59Google Scholar
  25. Schlag J, Schlag-Rey M (1984) Visuomotor functions of central thalamus in monkey. II. Unit activity related to visual events, targeting, and fixation. J Neurophysiol 51:1175–1195Google Scholar
  26. Segraves M, Goldberg ME (1987) Functional properties of corticotectal neurons in the monkey's frontal eye field. J Neurophysiol 58:1387–1419PubMedGoogle Scholar
  27. Selemon LD, Goldman-Rakic PS (1985) Longitudinal topography and interdigitation of corticostriatal projections in the rhesus moneky. J Neurosci 5:776–794PubMedGoogle Scholar
  28. Sparks DL (1986) Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol Rev 66:118–171PubMedGoogle Scholar
  29. Stadler MA (1995) The role of attention in implicit learning. J Exp Psychology: Learning, Mem, Cognition (in press)Google Scholar
  30. Stanton GB, Goldberg ME, Bruce CJ (1988a) Frontal eye field efferents in the macaque monkey. I. Subcortical pathways and topography of striatal and thalamic terminal fields. J Comp Neurol 271:473–492CrossRefPubMedGoogle Scholar
  31. Stanton GB, Goldberg ME, Bruce CJ (1988b) Frontal eye field efferents in the macaque monkey. II. Topography of terminal fields in idbrain and pons. J Comp Neurol 271:493–506CrossRefPubMedGoogle Scholar
  32. Van Essen DC (1979) Visual areas of the mammalian cerebral cortex. Annu Rev Neurosci 2:227–263CrossRefPubMedGoogle Scholar
  33. Wang D, Arbib M (1990) Complex temporal sequence learning based on short-term memory. Proc IEEE 78:1536–1543CrossRefGoogle Scholar
  34. Weitzenfield A (1991) NSL neural simulation language version 2. 1. (Technical Report 91-05). Center for Neural Engineering, University of Southern CaliforniaGoogle Scholar
  35. Yeterian EH, Pandya D (1991) Prefrontostriatal connections in relation to cortical architectonic organization in rhesus monkeys. J Comp Neurol 312:43–67CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag 1995

Authors and Affiliations

  • Peter F. Dominey
    • 1
  1. 1.Vision et Motricité, INSERM Unité 94BronFrance

Personalised recommendations