Particle Filter on Episode for Learning Decision Making Rule

  • Ryuichi Ueda
  • Kotaro Mizuta
  • Hiroshi Yamakawa
  • Hiroyuki Okada
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 531)


We propose a novel method, a particle filter on episode, for decision makings of agents in the real world. This method is used for simulating behavioral experiments of rodents as a workable model, and for decision making of actual robots. Recent studies on neuroscience suggest that hippocampus and its surroundings in brains of mammals are related to solve navigation problems, which are also essential in robotics. The hippocampus also handle memories and some parts of a brain utilize them for decision. The particle filter gives a calculation model of decision making based on memories. In this paper, we have verified that this method learns two kinds of tasks that have been frequently examined in behavioral experiments of rodents. Though the tasks have been different in character from each other, the algorithm has been able to make an actual robot take appropriate behavior in the both tasks with an identical parameter set.


Particle filter Decision making Learning Episodic memory 


  1. 1.
    Barbieri, R., et al.: An analysis of hippocampal spatio-temporal representations using a Bayesian algorithm for neural spike train decoding. IEEE Trans. Neural Syst. Rehabil. Eng. 13(2), 131–136 (2005)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Buzsáki, G., Moser, E.I.: Memory, navigation and theta rhythm in the hippocampal-entorhinal system. Nat. Neurosci. 16(2), 130–138 (2013)CrossRefGoogle Scholar
  3. 3.
    Dudchenko, P.A.: An overview of the tasks used to test working memory in rodents. Neurosci. Biobehav. Rev. 28(7), 699–709 (2004)CrossRefGoogle Scholar
  4. 4.
    Fox, D., et al.: Monte Carlo localization: efficient position estimation for mobile robots. In: Proceedings of AAAI. pp. 343–349 (1999)Google Scholar
  5. 5.
    Franz, M.O., Mallot, H.A.: Biomimetic robot navigation. Robot. Auton. Syst. 30, 133–153 (2000)CrossRefGoogle Scholar
  6. 6.
    Hafting, T., et al.: Microstructure of a spatial map in the entorhinal cortex. Nature 436, 801–806 (2005). AugCrossRefGoogle Scholar
  7. 7.
    Hargreaves, E.L., et al.: Major dissociation between medial and lateral entorhinal input to dorsal hippocampus. Science 308, 1792–1794 (2005). JuneCrossRefGoogle Scholar
  8. 8.
    Ito, H.T., et al.: A prefrontal-thalamo-hippocampal circuit for goal-directed spatial navigation. Nature 522, 50–55 (2015)CrossRefGoogle Scholar
  9. 9.
    Kitamura, T., et al.: Island cells control temporal association memory. Science 343(6173), 896–901 (2014)CrossRefGoogle Scholar
  10. 10.
    Lever, C., et al.: Boundary vector cells in the subiculum of the hippocampal formation. J. Neurosci. 29(31), 9771–9777 (2009)CrossRefGoogle Scholar
  11. 11.
    Milford, M., Schulz, R.: Principles of goal-directed spatial robot navigation in biomimetic models. Philos. Trans. R. Soc. B 369(1665), 2013484 (2014)Google Scholar
  12. 12.
    Milford, M.J., Wyeth, G.F.: Mapping a suburb with a single camera using a biologically inspired SLAM system. IEEE Trans. Robot. Autom. 24(5), 1038–1053 (2008)CrossRefGoogle Scholar
  13. 13.
    Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)CrossRefGoogle Scholar
  14. 14.
    Montemerlo, M.: FastSLAM: a factored solution to the simultaneous localization and mapping problem with unknown data association. Doctor Thesis, Carnegie Mellon University (2003)Google Scholar
  15. 15.
    Moser, E.I., Moser, M.B.: A metric for space. Hippocampus 18(12), 1142–1156 (2008)Google Scholar
  16. 16.
    O’keefe, J., Dostrovsky, J.: The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34(1), 171–175 (1971)Google Scholar
  17. 17.
    Ormoneit, D., Sen, Ś.: Kernel-based reinforcement learning. Mach. Learn. 49(2–3), 161–178 (2002)CrossRefzbMATHGoogle Scholar
  18. 18.
    Pastalkova, E., et al.: Internally generated cell assembly sequences in the rat Hippocampus. Science 321(5894), 1322–1327 (2008)CrossRefGoogle Scholar
  19. 19.
    Pfeiffer, B.E., Foster, D.J.: Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79 (2013)CrossRefGoogle Scholar
  20. 20.
    Shaw, C.L., et al.: The role of the medial prefrontal cortex in the acquisition, retention, and reversal of a tactile visuospatial conditional discrimination task. Behav. Brain Res. 236, 94–101 (2013)CrossRefGoogle Scholar
  21. 21.
    Solstad, T., et al.: Representation of geometric borders in the entorhinal cortex. Science 322(5909), 1865–1868 (2008)CrossRefGoogle Scholar
  22. 22.
    Spellman, T., et al.: Hippocampal-prefrontal input supports spatial encoding in working memory. Nature 522, 309–314 (2015)CrossRefGoogle Scholar
  23. 23.
    Suh, J.: Entorhinal cortex layer III input to the Hippocampus is crucial for temporal association memory. Science 334(9), 1415–1420 (2011)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)Google Scholar
  25. 25.
    Tesauro, G.: Temporal difference learning and TD-Gammon. Commun. ACM 38(3), 58–68 (1995)CrossRefGoogle Scholar
  26. 26.
    Thrun, S., et al.: Probabilistic Robotics. MIT Press (2005)Google Scholar
  27. 27.
    Ueda, R.: Generation of compensation behavior of autonomous robot for uncertainty of information with probabilistic flow control. Adv. Robot. 29(11), 721–734 (2015)CrossRefGoogle Scholar
  28. 28.
    Unemi, T., Saitoh, H.: Episode-based reinforcement learning—an instance-based approach for perceptual aliasing. In: Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, pp. 435–440 (1999)Google Scholar
  29. 29.
    Yamakawa, H.: Hippocampal formation mechanism will inspire frame generation for building an artificial general intelligence. In: Artificial General Intelligence, pp. 362–371 (2012)Google Scholar
  30. 30.
    Yamamoto, J., et al.: Successful execution of working memory linked to synchronized high-frequency gamma oscillations. Cell 157, 845–857 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Ryuichi Ueda
    • 1
  • Kotaro Mizuta
    • 2
  • Hiroshi Yamakawa
    • 3
  • Hiroyuki Okada
    • 4
  1. 1.Chiba Institute of TechnologyNarashino, ChibaJapan
  2. 2.Riken Brain Science InstituteSaitama, WakoJapan
  3. 3.Dowango Artificial Intelligence LaboratoryTokyoJapan
  4. 4.Graduate School of Brain Sciences, Tamagawa UniversityMachida, TokyoJapan

Personalised recommendations