Embedding Learning for Declarative Memories

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10249)


The major components of the brain’s declarative or explicit memory are semantic memory and episodic memory. Whereas semantic memory stores general factual knowledge, episodic memory stores events together with their temporal and spatial contexts. We present mathematical models for declarative memories where we consider semantic memory to be represented by triples and episodes to be represented as quadruples i.e., triples in time. E.g., (Jack, receivedDiagnosis, Diabetes, Jan1) states that Jack was diagnosed with diabetes on January 1. Both from a cognitive and a technical perspective, an interesting research question is how declarative data can efficiently be stored and semantically be decoded. We propose that a suitable data representation for episodic event data is a 4-way tensor with dimensions subject, predicate, object, and time. We demonstrate that the 4-way tensor can be decomposed, e.g., using a 4-way Tucker model, which permits semantic decoding of an event, as well as efficient storage. We also propose that semantic memory can be derived from the episodic model by a marginalization of the time dimension, which can be performed efficiently. We argue that the storage of episodic memory typically requires models with a high rank, whereas semantic memory can be modelled with a comparably lower rank. We analyse experimentally the relationship between episodic and semantic memory models and discuss potential relationships to the corresponding brain’s cognitive memories.


  1. 1.
    Atkinson, R.C., Shiffrin, R.M.: Human memory: a proposed system and its control processes. Psychol. Learn. Motiv. 2, 89–195 (1968). ElsevierCrossRefGoogle Scholar
  2. 2.
    Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.: DBpedia: a nucleus for a web of open data. In: Aberer, K., et al. (eds.) ASWC/ISWC -2007. LNCS, vol. 4825, pp. 722–735. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-76298-0_52CrossRefGoogle Scholar
  3. 3.
    Baddeley, A.D., Hitch, G., Bower, G.H.: The psychology of learning and motivation (1974)Google Scholar
  4. 4.
    Bartlett, F.C.: Remembering: A Study in Experimental and Social Psychology, vol. 14. Cambridge University Press, Cambridge (1995)CrossRefGoogle Scholar
  5. 5.
    Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: SIGMOD (2008)Google Scholar
  6. 6.
    Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems 26 (2013)Google Scholar
  7. 7.
    Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka Jr., E.R., Mitchell, T.M.: Toward an architecture for never-ending language learning. In: AAAI, vol. 5, p. 3 (2010)Google Scholar
  8. 8.
    Conway, M.A.: Episodic memories. Neuropsychologia 47(11), 2305–2313 (2009). ElsevierCrossRefGoogle Scholar
  9. 9.
    Ebbinghaus, H.: Über das Gedächtnis: Untersuchungen zur experimentellen Psychologie. Duncker & Humblot (1885)Google Scholar
  10. 10.
    Gazzaniga, M.S., Ivry, R.B., Mangun, G.R.: Cognitive Neuroscience: The Biology of the Mind. Norton, New York (2013)Google Scholar
  11. 11.
    Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS, vol. 9, pp. 249–256 (2010)Google Scholar
  12. 12.
    Gluck, M.A., Mercado, E., Myers, C.E.: Learning and Memory: From Brain to Behavior. Palgrave Macmillan, New York (2013)Google Scholar
  13. 13.
    Greenberg, D.L., Verfaellie, M.: Interdependence of episodic and semantic memory: evidence from neuropsychology. J. Int. Neuropsychological Soc. 16(05), 748–753 (2010). Cambridge Univ PressCrossRefGoogle Scholar
  14. 14.
    Gutierrez, C., Hurtado, C.A., Vaisman, A.: Introducing time into RDF. IEEE Trans. Knowl. Data Eng. 19(2) (2007). IEEECrossRefGoogle Scholar
  15. 15.
    Hoffart, J., Suchanek, F.M., Berberich, K., Lewis-Kelham, E., De Melo, G., Weikum, G.: Yago2: exploring and querying world knowledge in time, space, context, and many languages. In: WWW. ACM (2011)Google Scholar
  16. 16.
    Huth, A.G., de Heer, W.A., Griffiths, T.L., Theunissen, F.E., Gallant, J.L.: Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 532(7600), 453–458 (2016). Nature Publishing GroupCrossRefGoogle Scholar
  17. 17.
    Kesner, R.P., Rolls, E.T.: A computational theory of hippocampal function, and tests of the theory: new developments. Prog. Neurobiol. 79(1), 1–48 (2006). ElsevierCrossRefGoogle Scholar
  18. 18.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)
  19. 19.
    Krompaß, D., Jiang, X., Nickel, M., Tresp, V.: Probabilistic latent-factor database models. In: ECML PKDD (2014)Google Scholar
  20. 20.
    Kumar, A., Irsoy, O., Su, J., Bradbury, J., English, R., Pierce, B., Ondruska, P., Gulrajani, I., Socher, R.: Ask me anything: dynamic memory networks for natural language processing. arXiv:1506.07285 (2015)
  21. 21.
    Loftus, E., Ketcham, K.: The Myth of Repressed Memory: False Memories and Allegations of Sexual Abuse. Macmillan, New York (1996)Google Scholar
  22. 22.
    McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex. Psychol. Rev. 102(3), 419 (1995). American Psychological AssociationCrossRefGoogle Scholar
  23. 23.
    Morton, N.W.: Interactions between episodic and semantic memory. Technical report, Vanderbilt Computational Memory Lab (2013)Google Scholar
  24. 24.
    Nickel, M., Tresp, V., Kriegel, H.-P.: A three-way model for collective learning on multi-relational data. In: ICML (2011)Google Scholar
  25. 25.
    Nickel, M., Tresp, V., Kriegel, H.-P.: Factorizing YAGO: scalable machine learning for linked data. In: WWW (2012)Google Scholar
  26. 26.
    Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs: from multi-relational link prediction to automated knowledge graph construction. Proc. IEEE 104(1), 11–33 (2016). IEEECrossRefGoogle Scholar
  27. 27.
    Poon, H., Domingos, P.: Sum-product networks: a new deep architecture. In: ICCV (2011)Google Scholar
  28. 28.
    Quiroga, R.Q.: Concept cells: the building blocks of declarative memory functions. Nat. Rev. Neurosci. 13(8), 587–597 (2012). Nature Publishing GroupCrossRefGoogle Scholar
  29. 29.
    Roediger, H.L., McDermott, K.B.: Creating false memories: remembering words not presented in lists. J. Exp. Psychol. Learn. Mem. Cogn. 21(4), 803 (1995). American Psychological AssociationCrossRefGoogle Scholar
  30. 30.
    Rolls, E.T.: A computational theory of episodic memory formation in the hippocampus. Behav. Brain Res. 215(2), 180–196 (2010). ElsevierCrossRefGoogle Scholar
  31. 31.
    Rosenblatt, F.: The perceptron: a probabilistic model for information storage and organization in the brain. Psychol. Rev. 65(6), 386 (1958). American Psychological AssociationCrossRefGoogle Scholar
  32. 32.
    Singhal, A.: Introducing the knowledge graph: things, not strings, May 2012.
  33. 33.
    Socher, R., Gershman, S., Sederberg, P., Norman, K., Perotte, A.J., Blei, D.M.: A Bayesian analysis of dynamics in free recall. In: Advances in Neural Information Processing Systems (2009)Google Scholar
  34. 34.
    Squire, L.R.: Memory and Brain. Oxford University Press, New York (1987)Google Scholar
  35. 35.
    Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: WWW (2007)Google Scholar
  36. 36.
    Teyler, T.J., DiScenna, P.: The hippocampal memory indexing theory. Behav. Neurosci. 100(2), 147 (1986). American Psychological AssociationCrossRefGoogle Scholar
  37. 37.
    Tresp, V., Esteban, C., Yang, Y., Baier, S., Krompaß, D.: Learning with memory embeddings. arXiv:1511.07972 (2015)
  38. 38.
    Tulving, E.: Episodic and semantic memory 1. In: Organization of Memory. Academic, London (1972)Google Scholar
  39. 39.
    Yee, E., Chrysikou, E.G., Thompson-Schill, S.L.: The cognitive neuroscience of semantic memory. In: Oxford Handbook of Cognitive Neuroscience. Oxford University Press (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Siemens AG, Corporate TechnologyMunichGermany
  2. 2.Ludwig-Maximilians-Universität MünchenMunichGermany

Personalised recommendations