Design and integration of a spatio-temporal memory with emotional influences to categorize and recall the experiences of an autonomous mobile robot


Autonomous robots cohabiting with humans will have to achieve recurring tasks while adapting to the changing conditions of the world. A spatio-temporal memory categorizes the experiences of a robot to improve its ability to adapt to its environment. In this paper, we present a spatio-temporal (ST) memory model consisting of a cascade of two adaptive resonance theory (ART) networks: one to categorize spatial events and the other to extract temporal episodes from the robot’s experiences. Artificial emotions are used to dynamically modulate learning and recall of the ART networks based on how the robot is able to carry its task, using a simple model of artificial emotions. Once an episode is recalled, future events can be predicted and used to influence the intentions of the robot. Evaluation of our ST model is done using an autonomous robotic platform that has to deliver objects to people within an office area. Results demonstrate that our model can memorize and recall the experiences of a robot, and that emotional episodes are recalled more often, allowing the robot to use its memory of past experiences early on when repeating a task.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18


  1. Arras, K. O., Grzonka, S., Luber, M., & Burgard, W. (2008). Efficient people tracking in laser range data using a multi-hypothesis leg-tracker with adaptive occlusion probabilities. In Proceedings IEEE International conference on robotics and automation (pp. 1710–1715).

  2. Brachman, R. J. (2006). (AA)AI more than the sum of its parts. AI Magazine, 27(4), 19–34.

  3. Carpenter, G. A., & Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37(1), 54–115.

  4. Carpenter, G. A., Grossberg, S., & Rosen, D. B. (1991). Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system. Neural Networks, 4(6), 759–771.

  5. Dodd, W., & Gutierrez, R. (2005). The role of episodic memory and emotion in a cognitive robot. In Proceedings of IEEE International workshop on robot and human interactive communication (pp. 692–697).

  6. Ferland, F., Leconte, F., Tapus, A., & Michaud, F. (2014). An architecture for integrated episodic memory for adaptive robot behavior. In Workshop on artificial intelligence for human-robot interaction, association for the advancement of artificial intelligence (AAAI).

  7. Ferland, F., Létourneau, D., Aumont, A., Frémy, J., Legault, M. A., Lauria, M., et al. (2012). Natural interaction design of a humanoid robot. Journal of Human-Robot Interaction, 1(2), 118–134.

  8. Frank, T., Kraiss, K. F., & Kuhlen, T. (1998). Comparative analysis of Fuzzy ART and ART-2A network clustering performance. IEEE Transactions on Neural Networks, 9(3), 544–559. ID: 1.

  9. Grondin, F., Létourneau, D., Ferland, F., Rousseau, V., & Michaud, F. (2013). The ManyEars open framework: Microphone array open software and open hardware system for robotic applications. Autonomous Robots, 34(3), 217–232.

  10. Haikonen, P. O. (2007). Robot brains: Circuits and systems for conscious machines. Chichester: Wiley.

  11. Haikonen, P. O. (2012). Consciousness and robot sentience (Vol. 2). London: World Scientific Publishing Company.

  12. Hawkins, J. (2004). On intelligence. London: Macmillan.

  13. Jockel, S., Weser, M., Westhoff, D., & Zhang, J. (2008). Towards an episodic memory for cognitive robots. In Proceedings of 6th cognitive robotics workshop at 18th European conference on artificial intelligence (pp. 68–74).

  14. Kanthimathinathan, S., Prabhu, P. G., Kumar, S. J. P., & Kumar, P. A. (2004). Training goal keeper robot using emotion and history based learning method. In Proceedings of international conference on intelligent sensing and information processing, ID 1 (pp. 428–32).

  15. Komatsu, T., & Takeno, J. (2011). A conscious robot that expects emotions. In Proceedings of IEEE international conference on industrial technology (pp. 15–20).

  16. Kurzweil, R. (2012). How to Create a Mind: The Secret of Human Thought Revealed. Viking Penguin.

  17. Leconte, F., Ferland, F., & Michaud, F. (2014). Fusion adaptive resonance theory networks used as episodic memory for an autonomous robot. In Proceedings of the conference on artificial general intelligence.

  18. Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Proceedings of IEEE International conference on computer vision, vol. 2 (pp. 1150–1157).

  19. Markowitsch, H. J., & Staniloiu, A. (2012). Amnesic disorders. The Lancet, 380(9851), 1429–1440.

  20. McGaugh, J. L., & Roozendaal, B. (2002). Role of adrenal stress hormones in forming lasting memories in the brain. Current Opinion in Neurobiology, 12(2), 205–210.

  21. Michaud, F., Boissy, P., Labonté, D., Brière, S., Perreault, K., Corriveau, H., et al. (2010). Exploratory design and evaluation of a homecare teleassistive mobile robotic system. Mechatronics, Special Issue on Design and Control Methodologies in Telerobotics, 20(7), 751–766. doi:10.1016/j.mechatronics.2010.01.010.

  22. Michaud, F. (1998). Learning from history for behavior-based mobile robots in non-stationary environments. Special Issue on Learning in Autonomous Robots, Machine Learning/Autonomous Robots, 31(5), 141–167. 335–354.

  23. Nuxoll, A.M., & Laird, J.E. (2011). Enhancing intelligent agents with episodic memory. Cognitive Systems Research, 17–18, 34–48.

  24. Pfeifer, R., Bongard, J., & Grand, S. (2007). How the body shapes the way we think: A new view of intelligence. Cambridge: MIT Press.

  25. Phelps, E. A. (2004). Human emotion and memory: Interactions of the amygdala and hippocampal complex. Current Opinion in Neurobiology, 14(2), 198–202.

  26. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., & Ng, A.Y. (2009). ROS: An open-source robot operating system. In ICRA workshop on open source software.

  27. Raïevsky, C., & Michaud, F. (2009). Emotion generation based on a mismatch theory of emotions for situations agents, chap. 14. Information Science Reference, pp. 247–266.

  28. Rousseau, V., Ferland, F., Létourneau, D., & Michaud, F. (2013). orry to interrupt, but may I have your attention? Preliminary design and evaluation of autonomous engagement in HRI. Journal of Human-Robot Interaction, 2(3), 41–61.

  29. Squire, L. R. (2004). Memory systems of the brain: A brief history and current perspective. Neurobiology of Learning and Memory, 82(3), 171–177.

  30. Stachowicz, D., & Kruijff, G. (2011). Episodic-like memory for cognitive robots. IEEE Transactions on Autonomous Mental Development, 4(1), 1–16.

  31. Taylor, S. E., Vineyard, C. M., Healy, M. J., Caudell, T. P., Cohen, N. J., Watson, P., et al. (2009). Memory in silico: Building a neuromimetic episodic cognitive model. Proceedings of World Congress on Computer Science and Information Engineering, 5, 733–737.

  32. Thrun, S., Fox, D., Burgard, W., & Dellaert, F. (2001). Robust Monte Carlo localization for mobile robots. Artificial Intelligence, 128(1), 99–141.

  33. Tscherepanow, M., Kuhnel, S., & Riechers, S. (2012). Episodic clustering of data streams using a topology-learning neural network. In Proceedings of the european conference on artificial intelligence—workshop on active and incremental learning (pp. 24–29).

  34. Tulving, E. (1984). Precis of elements of episodic memory. Behavioral and Brain Sciences, 7(2), 223–268.

  35. Turk, M.A., & Pentland, A. (1991). Face recognition using eigenfaces. In: Proceedings IEEE on computer vision and pattern recognition (pp. 586–591).

  36. Valin, J. M., Michaud, F., & Rouat, J. (2006). Robust localization and tracking of simultaneous moving sound sources using beamforming and particle filtering. Robotics and Autonomous Systems, 55(3), 216–228.

  37. Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International Journal of Computer Vision, 57(2), 137–154.

  38. Wang, W., Subagdja, B., Tan, A. H., & Starzyk, J. A. (2010). A self-organizing approach to episodic memory modeling. In Proceedings of international joint conference on neural networks (pp. 1–8).

  39. Wang, W., Subagdja, B., Tan, A. H., & Starzyk, J. A. (2012). Neural modeling of episodic memory: Encoding, retrieval, and forgetting. IEEE Transactions on Neural Networks and Learning Systems, 23(10), 1574–1586.

Download references


The authors want to thank the volunteers who participated in the trials. This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).

Author information

Correspondence to Francis Leconte.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 185037 KB)

Supplementary material 1 (mp4 185037 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Leconte, F., Ferland, F. & Michaud, F. Design and integration of a spatio-temporal memory with emotional influences to categorize and recall the experiences of an autonomous mobile robot. Auton Robot 40, 831–848 (2016) doi:10.1007/s10514-015-9496-2

Download citation


  • Spatio-temporal memory
  • Adaptive resonance theory
  • Artificial emotions
  • Autonomous robots