ART neural network-based integration of episodic memory and semantic memory for task planning for robots

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.


Automated task planning for robots faces great challenges in that the sequences of events needed for a particular task are mostly required to be hard-coded. This can be a cumbersome process, especially, when the user wants a robot to learn a large number of similar tasks with different objects that are semantically related. We propose a novel approach of user preference-based integrated multi-memory model (pMM-ART). This approach focuses on exploiting a semantic hierarchy of objects alongside an episodic memory for enhancing the behavior of an autonomous agent. We analyze the functioning principle of the proposed model by teaching it a few distinct domestic tasks and observe that it is able to carry out a large number of similar tasks based on the semantic similarities between learned objects. We also demonstrate, via experiments using Mybot, our ability to reach those goals that are not possible without the integration of semantic knowledge with episodic memory.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8


  1. 1.


  1. Al-Moadhen, A., Qiu, R., Packianather, M., Ji, Z., & Setchi, R. (2013). Integrating robot task planner with common-sense knowledge base to improve the efficiency of planning. Procedia Computer Science, 22, 211–220.

  2. Benjamin, D. P., Lyons, D., & Lonsdale, D. (2004). Adapt: A cognitive architecture for robotics. In Proceedings of the international conference on cognitive modeling (pp. 337–338).

  3. Carpenter, G., Grossberg, S., & Rosen, D. (1991). Fuzzy art: Fast stable learning and categorization of analog patterns by an adaptive resonance system. Neural Networks, 4, 759–771.

  4. Carpenter, G. A., & Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37, 54–115.

  5. Cunningham, P. (1998). CBR: Strengths and weaknesses. In Proceedings of the 11th international conference on industrial and engineering applications of artificial intelligence and expert systems: Tasks and methods in applied artificial intelligence (pp. 517–524).

  6. Dayoub, F., Duckett, T., & Cielniak, G. (2010). Toward an object-based semantic memory for long-term operation of mobile service robots. In IROS workshop on semantic mapping and autonomous knowledge acquisition.

  7. Galindo, C., Fernandez-Madrigal, J., Gonzalez, J., & Saffiotti, A. (2008). Robot task planning using semantic maps. Robotics and Autonomous Systems, 56(11), 955–966.

  8. Gao, S., & Tan, A. H. (2014). A multi-memory modeling approach. In Proceedings of the international joint conference on neural networks (pp. 1542–1548).

  9. Greenberg, D. L., & Verfaellie, M. (2015). Interdependence of episodic and semantic memory: Evidence from neuropsychology. Journal of the International Neuropsychological Society, 16, 748–753.

  10. Hawkins, J., George, D., & Niemasik, J. (2009). Sequence memory for prediction, inference and behaviour. Philosophical Transactions of the Royal Society B, 364, 1203–1209.

  11. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9, 1735–1780.

  12. Irish, M., & Piguet, O. (2013). The pivotal role of semantic memory in remembering the past and imagining the future. Frontiers in Behavioral Neuroscience.

  13. Jeong, I. B., Lee, S. J., & Kim, J. H. (2015). RRT\(\ast \)-quick: A motion planning algorithm with faster convergence rate. In J. H. Kim, W. Yang, J. Jo, P. Sincak, & H. Myung (Eds.), Robot intelligence technology and applications 3. Advances in intelligent systems and computing (Vol. 345). Cham: Springer.

  14. Ji, Z., Qiu, R., Noyvirt, A., Soroka, A., Packianather, M., Setchi, R., et al. (2012). Towards automated task planning for service robots using semantic knowledge representation. In Proceedings of the IEEE international conference on industrial information (pp. 1194–1201).

  15. Kim, J. H., Choi, S. H., Park, I. W., & Zaheer, S. A. (2013). Intelligence technology for robots that think. IEEE Computational Intelligence Magazine, 8, 70–84.

  16. Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987). Soar: An architecture for general intelligence. Artificial Intelligence, 33(1), 1–64.

  17. Leconte, F., Ferland, F., & Michaud, F. (2015). Design and integration of a spatio-temporal memory with emotional influences to categorize and recall the experiences of an autonomous mobile robot. Autonomous Robots, 40, 831–848.

  18. Levine, B., Turner, G. R., Tisserand, D., Hevanor, S. J., Graham, S. J., & McIntosh, A. R. (2004). The functional neuroanatomy of episodic and semantic autobiographical remembering: A prospective functional mri study. Journal of Cognitive Neuroscience, 16, 1633–1646.

  19. McRae, K., & Jones, M. N. (2013). The Oxford handbook of cognitive psychology. Oxford: Oxford University Press (Chap Semantic memory).

  20. Mermillod, M., Bugaiska, A., & Bonin, P. (2013). The stability–plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4, 504.

  21. Nasir, J., & Kim, J. H. (2016). User preference-based integrated multi-memory neural model for improving the cognitive abilities of autonomous robots. Master’s thesis, Korea Advanced Institute of Science and Technology.

  22. Nasir, J., Yoo, Y. H., Kim, D. H., & Kim, J. H. (2018). User preference-based dual-memory neural model with memory consolidation approach. IEEE Transactions on Neural Networks and Learning Systems, 29(6), 2294–2308.

  23. Nuxoll, A. M., & Laird, J. E. (2007). Extending cognitive architecture with episodic memory. In Proceedings of the 22nd national conference on artificial intelligence AAAI’07(Vol. 2, pp. 1560–1565). New York: AAAI Press.

  24. Nuxoll, A. M., & Laird, J. E. (2012). Enhancing intelligent agents with episodic memory. Cognitive Systems Research, 17–18, 34–48.

  25. Riesbeck, C. K., & Schank, R. (1989). Inside case-based reasoning. Hillsdale: L. Erlbaum Associates Inc.

  26. Rogers III, J. G., & Christensen, H. I. (2013). Robot planning with a semantic map. In IEEE international conference on robotics and automation (pp. 2239–2244).

  27. Shapiro, S., & Bona, J. P. (2009). The glair cognitive architecture. International Journal of Machine Consciousness, 2(2), 307–332.

  28. Stachowicz, D., & Kruijff, G. M. (2012). Episodic-like memory for cognitive robots. IEEE Transactions on Autonomous Mental Development, 4(1), 1–16.

  29. Starzyk, J. A., & He, H. (2007). Anticipation-based temporal sequences learning in hierarchical structure. IEEE Transactions on Neural Networks, 18(2), 344–358.

  30. Starzyk, J. A., & He, H. (2009). Spatio-temporal memories for machine learning: A long-term memory organization. IEEE Transactions on Neural Networks, 20, 768–780.

  31. Subagdja, B., & Tan, A. H. (2012). iFALCON: A neural architecture for hierarchical planning. Neurocomputing, 86, 124–139.

  32. Sucan, I. A., & Chitta, S. (2011). Moveit!

  33. Tan, A. H., Carpenter, G. A., & Grossberg, S. (2007). Intelligence through interaction: Towards a unified theory for learning. In D. Liu, S. Fei, Z. G. Hou, H. Zhang, & C. Sun (Eds.), Advances in neural networks—ISNN 2007 (pp. 1094–1103). Berlin: Springer.

  34. Tan, A. H., Feng, Y. H., & Ong, Y. S. (2010). A self-organizing neural architecture integrating desire, intention and reinforcement learning. Neurocomputing, 73(7–9), 1465–1477.

  35. Taylor, S. E., Vineyard, C. M., Healy, M. J., Caudell, T. P., Cohen, N. J., Watson, P., et al. (2009). Memory in silico: Building a neuromimetic episodic cognitive model. In Proceedings of world congress on computer science and information engineering (Vol. 5, pp. 733–737).

  36. Tscherepanow, M. (2010) Topoart: A topology learning hierarchical art network. In Proceedings of the international conference on artificial neural networks (pp. 157–167).

  37. Tscherepanow, M., Kuhnel, S., & Riechers, S. (2012). Episodic clustering of data streams using a topology-learning neural network. In Proceedings of the European conference on artificial intelligence—Workshop on active and incremental learning (pp. 22–24).

  38. Tulving, E. (1972). Episodic and semantic memory. New York: Academic.

  39. Tulving, E. (1983). Elements of episodic memory. New York: Oxford University Press.

  40. Tulving, E. (2002). Episodic memory: From mind to brain. Annual Review of Psychology, 53, 1–25.

  41. Veiga, T. A., Miraldo, P., Ventura, R., & Lima, P. U. (2016). Efficient object search for mobile robots in dynamic environments: Semantic map as an input for the decision maker. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 2745–2750).

  42. Wang, D., & Arbib, M. A. (1990). Complex temporal sequence learning based on short-term memory. Proceedings of the IEEE, 78(9), 1536–1543.

  43. Wang, D., & Yuwono, B. (1995). Anticipation-based temporal pattern generation. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 25, 615–628.

  44. Wang, L. (1999). Multi-associative neural networks and their application to learning and retrieving complex spatio-temporal sequences. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 29, 73–82.

  45. Wang, W., Subagdja, B., Tan, A. H., & Starzyk, J. A. (2012a). Neural modeling of episodic memory: Encoding, retrieval, and forgetting. IEEE Transactions on Neural Networks and Learning Systems, 23(10), 1574–1586.

  46. Wang, W., Subagdja, B., Tan, A. H., & Tan, Y. (2012b). A self-organizing multi-memory system for autonomous agents. In Proceedings of the international joint conference on neural networks (pp. 252–258).

  47. Wang, W., Tan, A. H., & Teow, L. (2017). Semantic memory modeling and memory interaction in learning agents. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(11), 2882–2895.

  48. Wu, C., Lenz, I., & Saxena, A. (2014). Hierarchical semantic labeling for task-relevant RGB-D perception. In Robotics: Science and systems.

Download references


This work was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korea (MSIP) (No. NRF-2014R1A2A1A10051551) and the Technology Innovation Program, 10045252, funded by the Korea MOTIE. The authors would like to thank Yong-Ho Yoo for his guidance during experiments on Mybot. The authors would also like to thank Jennifer Olsen, a post-doc at Computer–Human Interaction in Learning and Instruction Lab., for her feedback on the draft.

Author information

Correspondence to Jauwairia Nasir.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 33522 KB)

Supplementary material 1 (mp4 33522 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Nasir, J., Kim, D. & Kim, J. ART neural network-based integration of episodic memory and semantic memory for task planning for robots. Auton Robot 43, 2163–2182 (2019).

Download citation


  • Adaptive resonance theory
  • Task planning
  • Cognition
  • Semantic memory
  • Episodic memory
  • User preference