Skip to main content

Advertisement

SpringerLink
Log in

Path planning versus cue responding: a bio-inspired model of switching between navigation strategies

  • Original Paper
  • Published:
Biological Cybernetics Aims and scope Submit manuscript

Abstract

In this article, we describe a new computational model of switching between path-planning and cue-guided navigation strategies. It is based on three main assumptions: (i) the strategies are mediated by separate memory systems that learn independently and in parallel; (ii) the learning algorithms are different in the two memory systems—the cue-guided strategy uses a temporal-difference (TD) learning rule to approach a visible goal, whereas the path-planning strategy relies on a place-cell-based graph-search algorithm to learn the location of a hidden goal; (iii) a strategy selection mechanism uses TD-learning rule to choose the most successful strategy based on past experience. We propose a novel criterion for strategy selection based on the directions of goal-oriented movements suggested by the different strategies. We show that the selection criterion based on this “common currency” is capable of choosing the best among TD-learning and planning strategies and can be used to solve navigational tasks in continuous state and action spaces. The model has been successfully applied to reproduce rat behavior in two water-maze tasks in which the two strategies were shown to interact. The model was used to analyze competitive and cooperative interactions between different strategies during these tasks as well as relative influence of different types of sensory cues.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Arleo A, Gerstner W (2000) Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol Cybern 83(3): 287–299

    Article  CAS  PubMed  Google Scholar 

  • Arleo A, Rondi-Reig L (2007) Multimodal sensory integration and concurrent navigation strategies for spatial cognition in real and artificial organisms. J Integr Neurosci 6(3): 327–366

    Article  PubMed  Google Scholar 

  • Barrera A, Weitzenfeld A (2007) Bio-inspired model of robot spatial cognition: topological place recognition and target learning. In: CIRA, pp 61–66

  • Blaisdell A (2009) The role of associative processes in spatial, temporal, and causal cognition. In: Watanabe SB, Blaisdell AP, Huber L, Young A (eds) Rational animals, irrational humans. Keio University Press, Tokyo, pp 153–172

    Google Scholar 

  • Brown M, Sharp P (1995) Simulation of spatial learning in the Morris water maze by a neural network model of the hippocampal formation and nucleus accumbens. Hippocampus 5(3): 171–188

    Article  CAS  PubMed  Google Scholar 

  • Burgess N (2008) Spatial cognition and the brain. Ann N Y Acad Sci 1124: 77–97

    Article  PubMed  Google Scholar 

  • Burnod Y (1991) Organizational levels of the cerebral cortex: an integrated model. Acta Biotheor 39(3–4): 351–361

    Article  CAS  PubMed  Google Scholar 

  • Canal C, Stutz S, Gold P (2005) Glucose injections into the dorsal hippocampus or dorsolateral striatum of rats prior to T-maze training: modulation of learning rates and strategy selection. Learn Mem 12(4): 367–374

    Article  PubMed  Google Scholar 

  • Chamizo V (2003) Acquisition of knowledge about spatial location: assessing the generality of the mechanism of learning. Q J Exp Psychol 56(1): 102–113

    Article  CAS  Google Scholar 

  • Chang Q, Gold PE (2003) Switching memory systems during learning: changes in patterns of brain acetylcholine release in the hippocampus and striatum in rats. J Neurosci 23(7): 3001

    CAS  PubMed  Google Scholar 

  • Chang Q, Gold PE (2004) Inactivation of dorsolateral striatum impairs acquisition of response learning in cue-deficient, but not cue-available, conditions. Behav Neurosci 118(2): 383–388

    Article  PubMed  Google Scholar 

  • Chavarriaga R, Strösslin T, Sheynikhovich D, Gerstner W (2005) A computational model of parallel navigation systems in rodents. Neuroinformatics 3(3): 223–242

    Article  PubMed  Google Scholar 

  • Conde F, Maire-Lepoivre E, Audinat E, Crepel F (1995) Afferent connections of the medial frontal cortex of the rat. II. Cortical and subcortical afferents. J Comp Neurol 352(4): 567–593

    Article  CAS  PubMed  Google Scholar 

  • Cuperlier N, Quoy M, Gaussier P (2007) Neurobiologically inspired mobile robot navigation and planning. Front Neurorobotics 1: 1–15

    Google Scholar 

  • Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12): 1704–1711

    Article  CAS  PubMed  Google Scholar 

  • Descarries L, Lemay B, Doucet G, Berger B (1987) Regional and laminar density of the dopamine innervation in adult rat cerebral cortex. Neuroscience 21(3): 807–824

    Article  CAS  PubMed  Google Scholar 

  • Devan B, White N (1999) Parallel information processing in the dorsal striatum: relation to hippocampal function. J Neurosci 19(7): 2789–2798

    CAS  PubMed  Google Scholar 

  • Devan B, McDonald R, White N (1999) Effects of medial and lateral caudate-putamen lesions on place-and cue-guided behaviors in the water maze: relation to thigmotaxis. Behav Brain Res 100(1–2): 5–14

    Article  CAS  PubMed  Google Scholar 

  • Dijkstra E (1959) A note on two problems in connection with graphs. Numer Math 1(269–270): 269–271

    Article  Google Scholar 

  • Doeller CF, Burgess N (2008) Distinct error-correcting and incidental learning of location relative to landmarks and boundaries. Proc Natl Acad Sci USA 105(15): 5909–5914

    Article  CAS  PubMed  Google Scholar 

  • Doeller CF, King JA, Burgess N (2008) Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory. Proc Natl Acad Sci USA 105(15): 5915–5920

    Article  CAS  PubMed  Google Scholar 

  • Dolle L, Khamassi M, Girard B, Guillot A, Chavarriaga R (2008) Analyzing interactions between navigation strategies using a computational model of action selection. LNAI 5248: 71–86

    Google Scholar 

  • Foster DJ, Morris RG, Dayan P (2000) A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10(1): 1–16

    Article  CAS  PubMed  Google Scholar 

  • Franz MO, Mallot HA (2000) Biomimetic robot navigation. Rob Auton Syst 30(1): 133–153

    Article  Google Scholar 

  • Gibson B, Shettleworth S (2003) Competition among spatial cues in a naturalistic food-carrying task. Learn Behav 31(2): 143–159

    PubMed  Google Scholar 

  • Gibson B, Shettleworth S (2005) Place versus response learning revisited: tests of blocking on the radial maze. Behav Neurosci 119(2): 567–586

    Article  PubMed  Google Scholar 

  • Girard B, Filliat D, Meyer J, Berthoz A, Guillot A (2005) Integration of navigation and action selection functionalities in a computational model of cortico-basal-thalamo-cortical loops. Adapt Behav 13(2): 115–130

    Article  Google Scholar 

  • Gold P (2004) Coordination of multiple memory systems. Neurobiol Learn Mem 82(3): 230–242

    Article  PubMed  Google Scholar 

  • Grahn J, Parkinson J, Owen A (2008) The cognitive functions of the caudate nucleus. Prog Neurobiol 86(3): 141–155

    Article  PubMed  Google Scholar 

  • Granon S, Poucet B (1995) Medial prefrontal lesions in the rat and spatial navigation: evidence for impaired planning. Behav Neurosci 109(3): 474–484

    Article  CAS  PubMed  Google Scholar 

  • Groenewegen H, Berendse H, Meredith G, Haber S, Voorn P, Wolters J, Lohman A (1991) The mesolimbic dopamine system: from motivation to action. In: Willner P, Scheel-Kriiger J (eds) Functional anatomy of the ventral, limbic system-innervated striatum. Wiley, Chichester, pp 19–59

    Google Scholar 

  • Guazzelli A, Corbacho F, Bota M, Arbib M (1998) Affordances, motivation, and the world graph theory. Adapt Behav 6(3): 435– 471

    Article  Google Scholar 

  • Hamilton D, Rosenfelt C, Whishaw I (2004) Sequential control of navigation by locale and taxon cues in the morris water task. Behav Brain Res 154(2): 385–397

    Article  PubMed  Google Scholar 

  • Hartley T, Burgess N (2005) Complementary memory systems: competition, cooperation and compensation. Trends Neurosci 28(4): 169–170

    Article  CAS  PubMed  Google Scholar 

  • Hasselmo ME (2005) A model of prefrontal cortical mechanisms for goal-directed behavior. J Cogn Neurosci 17(7): 1115–1129

    Article  PubMed  Google Scholar 

  • Jankowski J, Scheef L, Hüppe C, Boecker H (2009) Distinct striatal regions for planning and executing novel and automated movement sequences. Neuroimage 44(4): 1369–1379

    Article  CAS  PubMed  Google Scholar 

  • Kelly D, Gibson B (2007) Spatial navigation: spatial learning in real and virtual environments. Comp Cogn Behav Rev 2: 111–124

    Google Scholar 

  • Khamassi M (2007) Complementary roles of the rat prefrontal cortex and striatum in reward-based learning and shifting navigation strategies. PhD thesis, University Paris 6

  • Kim J, Baxter M (2001) Multiple brain-memory systems: the whole does not equal the sum of its parts. Trends Neurosci 24(6): 324–330

    Article  CAS  PubMed  Google Scholar 

  • Leising K, Blaisdell A (2009) Associative basis of landmark learning and integration in vertebrates. Comp Cogn Behav Rev 4: 80–102

    PubMed  Google Scholar 

  • Martinet LE, Passot JB, Fouque B, Meyer JA, Arleo A (2008) Map-based spatial navigation: a cortical column model for action planning. LNAI 5248: 39–55

    Google Scholar 

  • McDonald R, White N (1993) A triple dissociation of memory systems: hippocampus, amygdala, and dorsal striatum. Behav Neurosci 107(1): 3–22

    Article  CAS  PubMed  Google Scholar 

  • McDonald R, White N (1994) Parallel information processing in the water maze: evidence for independent memory systems involving dorsal striatum and hippocampus. Behav Neural Biol 61(3): 260–270

    Article  CAS  PubMed  Google Scholar 

  • McDonald R, Devan B, Hong N (2004) Multiple memory systems: the power of interactions. Neurobiol Learn Mem 82(3): 333–346

    Article  PubMed  Google Scholar 

  • Mizumori S (2008) Hippocampal place fields. Oxford University Press, USA

    Book  Google Scholar 

  • O’Keefe J, Dostrovsky J (1971) The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res 34(1): 171–175

    Article  PubMed  Google Scholar 

  • O’Keefe J, Nadel L (1978) The hippocampus as a cognitive map. Oxford University Press, Oxford

    Google Scholar 

  • Packard M, McGaugh J (1992) Double dissociation of fornix and caudate nucleus lesions on acquisition of two water maze tasks: further evidence for multiple memory systems. Behav Neurosci 106(3): 439–446

    Article  CAS  PubMed  Google Scholar 

  • Packard M, McGaugh J (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol Learn Mem 65(1): 65–72

    Article  CAS  PubMed  Google Scholar 

  • Packard M, Hirsh R, White N (1989) Differential effects of fornix and caudate nucleus lesions on two radial maze tasks: evidence for multiple memory systems. J Neurosci 9: 1465–1472

    CAS  PubMed  Google Scholar 

  • Pearce J (2009) The 36th Sir Frederick Bartlett Lecture: an associative analysis of spatial learning. Q J Exp Psychol 62(9): 1665–1684

    Article  Google Scholar 

  • Pearce J, Roberts A, Good M (1998) Hippocampal lesions disrupt navigation based on cognitive maps but not heading vectors. Nature 396(6706): 75–77

    Article  CAS  PubMed  Google Scholar 

  • Pouzet B, Zhang W, Feldon J, Rawlins J (2002) Hippocampal lesioned rats are able to learn a spatial position using non-spatial strategies. Behav Brain Res 133(2): 279–291

    Article  PubMed  Google Scholar 

  • Ragozzino M, Detrick S, Kesner R (1999) Involvement of the prelimbic-infralimbic areas of the rodent prefontal cortex in behavioral flexibility for place and response learning. J Neurosci 19(11): 4585–4594

    CAS  PubMed  Google Scholar 

  • Redish A (1999) Beyond the cognitive map: from place cells to episodic memory. The MIT Press, Cambridge

    Google Scholar 

  • Rescorla R, Wagner A (1972) A theory of pavlovian conditioning: the effectiveness of reinforcement and non-reinforcement. In: Black A, Prokasy W (eds) Classical conditioning II: current research and theory. Appleton-Century-Crofts, New York, pp 64–69

    Google Scholar 

  • Rich E, Shapiro M (2007) Prelimbic/infralimbic inactivation impairs memory for multiple task switches, but not flexible selection of familiar tasks. J Neurosci 27(17): 4747

    Article  CAS  PubMed  Google Scholar 

  • Rich E, Shapiro M (2009) Rat prefrontal cortical neurons selectively code strategy switches. J Neurosci 29(22): 7208–7219

    Article  CAS  PubMed  Google Scholar 

  • Roberts A, Pearce J (1999) Blocking in the Morris swimming pool. J Exp Psychol Anim Behav Process 25(2): 225–235

    Article  CAS  PubMed  Google Scholar 

  • Save E, Poucet B (2000) Involvement of the hippocampus and associative parietal cortex in the use of proximal and distal landmarks for navigation. Behav Brain Res 109(2): 195–206

    Article  CAS  PubMed  Google Scholar 

  • Sheynikhovich D, Chavarriaga R, Strösslin T, Arleo A, Gerstner W (2009) Is there a geometric module for spatial orientation? Insights from a rodent navigation model. Psychol Rev 116(3): 540–566

    Article  PubMed  Google Scholar 

  • Skaggs W, Knierim J, Kudrimoti H, McNaughton B (1995) A model of the neural basis of the rat’s sense of direction. Adv Neural Inf Process Syst 7: 173–182

    CAS  PubMed  Google Scholar 

  • Stahlman W, Blaisdell A (2009) Blocking of spatial control by landmarks in rats. Behav Processes 81(1): 114–118

    Article  PubMed  Google Scholar 

  • Strösslin T, Sheynikhovich D, Chavarriaga R, Gerstner W (2005) Robust self-localisation and navigation based on hippocampal place cells. Neural Netw 18(9): 1125–1140

    Article  PubMed  Google Scholar 

  • Sutton R, Barto A (1998) Reinforcement learning: an introduction. Bradford Book. The MIT Press, Cambridge

    Google Scholar 

  • Taube JS, Muller RU, Ranck JB Jr (1990) Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis. J Neurosci 10(2): 420

    CAS  PubMed  Google Scholar 

  • Touretzky D, Redish A (1996) Theory of rodent navigation based on interacting representations of space. Hippocampus 6(3): 247–270

    Article  CAS  PubMed  Google Scholar 

  • Uchibe E, Doya K (2005) Reinforcement learning with multiple heterogeneous modules: a framework for developmental robot learning. In: The 4th international conference on development and learning. IEEE Computer Society Press, pp 87–92

  • Ujfalussy B, Eros P, Somogyvari Z, Kiss T (2008) Episodes in space: a modelling study of hippocampal place representation. LNAI 5040: 123–136

    Google Scholar 

  • Voermans N, Petersson K, Daudey L, Weber B, Van Spaendonck K, Kremer H, Fernández G (2004) Interaction between the human hippocampus and the caudate nucleus during route recognition. Neuron 43(3): 427–435

    Article  CAS  PubMed  Google Scholar 

  • White N (2004) The role of stimulus ambiguity and movement in spatial navigation: a multiple memory systems analysis of location discrimination. Neurobiol Learn Mem 82: 216–229

    Article  PubMed  Google Scholar 

  • White N (2009) Some highlights of research on the effects of caudate nucleus lesions over the past 200 years. Behav Brain Res 199(1): 3–23

    Article  PubMed  Google Scholar 

  • White N, McDonald R (2002) Multiple parallel memory systems in the brain of the rat. Neurobiol Learn Mem 77: 125–184

    Article  PubMed  Google Scholar 

  • Yin H, Knowlton B (2004) Contributions of striatal subregions to place and response learning. Learn Mem 11(4): 459–463

    Article  PubMed  Google Scholar 

  • Young J, Shapiro M (2009) Double dissociation and hierarchical organization of strategy switches and reversals in the rat PFC. Behav Neurosci 123(5): 1028–1035

    Article  PubMed  Google Scholar 

  • Zhang K (1996) Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. J Neurosci 16(6): 2112

    CAS  PubMed  Google Scholar 

Download references

Author information

Author notes

  1. Denis Sheynikhovich

    Present address: Neurobiology of Adaptive Processes UPMC, CNRS UMR 7102, 9 quai St. Bernard, 75005, Paris, France

Authors and Affiliations

  1. Institut des Systèmes Intelligents et de Robotique, UPMC CNRS UMR 7222, 4 Place Jussieu, 75252, Paris Cedex 05, France

    Laurent Dollé, Denis Sheynikhovich, Benoît Girard & Agnès Guillot

  2. Defitech chair on Non-Invasive Brain-Computer Interface (CNBI), Ecole Polytechnique Fédérale de Lausanne (EPFL), 1015, Lausanne, Switzerland

    Ricardo Chavarriaga

Authors
  1. Laurent Dollé
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Denis Sheynikhovich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benoît Girard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ricardo Chavarriaga
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Agnès Guillot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laurent Dollé.

Additional information

Laurent Dollé, Denis Sheynikhovich—First authorship shared.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Dollé, L., Sheynikhovich, D., Girard, B. et al. Path planning versus cue responding: a bio-inspired model of switching between navigation strategies. Biol Cybern 103, 299–317 (2010). https://doi.org/10.1007/s00422-010-0400-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00422-010-0400-z

Keywords

Search

Navigation