Evolving Distributed Representations for Language with Self-Organizing Maps

  • Simon D. Levy
  • Simon Kirby
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4211)


We present a neural-competitive learning model of language evolution in which several symbol sequences compete to signify a given propositional meaning. Both symbol sequences and propositional meanings are represented by high-dimensional vectors of real numbers. A neural network learns to map between the distributed representations of the symbol sequences and the distributed representations of the propositions. Unlike previous neural network models of language evolution, our model uses a Kohonen Self-Organizing Map with unsupervised learning, thereby avoiding the computational slowdown and biological implausibility of back-propagation networks and the lack of scalability associated with Hebbian-learning networks. After several evolutionary generations, the network develops systematically regular mappings between meanings and sequences, of the sort traditionally associated with symbolic grammars. Because of the potential of neural-like representations for addressing the symbol-grounding problem, this sort of model holds a good deal of promise as a new explanatory mechanism for both language evolution and acquisition.


Weight Vector Language Evolution Word Order Latent Semantic Analysis Symbol Sequence 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Harnad, S.: Grounding symbols in the analog world with neural nets. Think 2(1), 12–78 (1993)Google Scholar
  2. 2.
    Searle, J.: Minds, brains, and programs. Behavioral and Brain Sciences 3 (1980)Google Scholar
  3. 3.
    Batali, J.: Computational simulations of the emergence of grammar. In: Hurford, J., Studdert-Kennedy, M., Knight, C. (eds.) Approaches to the Evolution of Language: Social and Cognitive Bases. Cambridge University Press, Cambridge (1998)Google Scholar
  4. 4.
    Rumelhart, D., Hinton, G., Williams, R.: Learning internal representation by error propagation. In: Rumelhart, D., McClelland, J. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1. MIT Press, Cambridge (1986)Google Scholar
  5. 5.
    Smith, K.: The cultural evolution of communication in a population of neural networks. Connection Science 14(1), 65–84 (2002)CrossRefGoogle Scholar
  6. 6.
    Fodor, J.: The Language of Thought. Crowell, New York (1975)Google Scholar
  7. 7.
    Grossberg, S.: Competitive learning: from interactive activation to adaptive resonance. In: Connectionist models and their implications: readings from cognitive science, pp. 243–283. Ablex Publishing Corp., Norwood (1988)Google Scholar
  8. 8.
    McClelland, J., Rumelhart, D., Hinton, G.: The appeal of parallel distributed processing. In: Rumelhart, D., McClelland, J. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, MIT Press, Cambridge (1986)Google Scholar
  9. 9.
    Smith, K., Brighton, H., Kirby, S.: Complex systems in language evolution: the cultural emergence of compositional structure. Advances in Complex Systems 6(4), 537–558 (2003)CrossRefGoogle Scholar
  10. 10.
    Hinton, G.: Distributed representations. Technical Report CMU-CS-84-157, Computer Science Department, Carnegie Mellon University (1984)Google Scholar
  11. 11.
    Elman, J.: Finding structure in time. Cognitive Science 14, 179–211 (1990)CrossRefGoogle Scholar
  12. 12.
    Landauer, T.K., Dumais, S.T.: A solution to plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review 104, 211–240 (1997)CrossRefGoogle Scholar
  13. 13.
    Gärdenfors, P.: Conceptual Spaces: The Geometry of Thought. MIT Press, Cambridge (2000)Google Scholar
  14. 14.
    Steedman, M.: Connectionist sentence processing in perspective. Cognitive Science 23(4), 615–634 (1999)CrossRefGoogle Scholar
  15. 15.
    Plate, T.A.: Holographic Reduced Representation: Distributed Representation for Cognitive Science. CSLI Publications (2003)Google Scholar
  16. 16.
    Kanerva, P.: The binary spatter code for encoding concepts at many levels. In: Marinaro, M., Morasso, P. (eds.) ICANN 1994: Proceedings of International Conference on Artificial Neural Networks, vol. 1, pp. 226–229. Springer, London (1994)Google Scholar
  17. 17.
    Rachkovskij, D.A., Kussul, E.M.: Binding and normalization of binary sparse distributed representations by context-dependent thinning. Neural Computation 13(2), 411–452 (2001)MATHCrossRefGoogle Scholar
  18. 18.
    Gayler, R.: Multiplicative binding, representation operators, and analogy. In: Holyoak, K., Gentner, D., Kokinov, B. (eds.) Advances in Analogy Research: Integration of Theory and Data from the Cognitive, Computational, and Neural Sciences. New Bulgarian University, Sofia, Bulgaria, p. 405 (1998)Google Scholar
  19. 19.
    Pollack, J.: Recursive distributed representations. Artifical Intelligence 36, 77–105 (1990)CrossRefGoogle Scholar
  20. 20.
    Smolensky, P.: Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence 46, 159–216 (1990)MATHCrossRefMathSciNetGoogle Scholar
  21. 21.
    Kohonen, T.: Self-Organizing Maps, 3rd edn. Springer, Secaucus (2001)MATHGoogle Scholar
  22. 22.
    VanHulle, M.: Faithful Representations and Topographic Maps. Wiley-Interscience, New York (1990)Google Scholar
  23. 23.
    Brighton, H., Kirby, S.: Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial Life 12(2), 229–242 (2006)CrossRefGoogle Scholar
  24. 24.
    Kirby, S.: Learning, bottlenecks and the evolution of recursive syntax. In: Briscoe, T. (ed.) Linguistic Evolution through Language Acquisition: Formal and Computational Models, Cambridge University Press, Cambridge (2002)Google Scholar
  25. 25.
    MacLennan, B.: Synthetic ethology: An approach to the study of communication. In: Langton, C., Taylor, C., Farmer, D., Rasmussen, S. (eds.) Artificial Life II, pp. 631–658. Addison-Wesley, Redwood City (1992)Google Scholar
  26. 26.
    Werner, G., Dyer, M.: Evolution of communication in artificial organisms. In: Langton, C., Taylor, C., Farmer, D., Rasmussen, S. (eds.) Artificial Life II, pp. 659–687. Addison-Wesley, Redwood City (1992)Google Scholar
  27. 27.
    Briscoe, T.: Grammatical acquisition: Inductive bias and coevolution of language and the language acquisition device. Language 76(2), 245–296 (2000)CrossRefGoogle Scholar
  28. 28.
    Chomsky, N.: Language and Mind. Harcourt Brace Jovanovich, San Diego (1972)Google Scholar
  29. 29.
    Smith, K.: Natural selection and cultural selection in the evolution of communication. Adaptive Behavior 10(1), 25–44 (2002)CrossRefGoogle Scholar
  30. 30.
    Chomsky, N.: Rules and Representations. Basil Blackwell, Oxford (1980)Google Scholar
  31. 31.
    Croft, W.: Explaining language change: an evolutionary approach. Longman, Harlow, Essex (2000)Google Scholar
  32. 32.
    Lewandowsky, S., Murdock, B.: Memory for serial order. Psychological Review 96(1), 25–27 (1989)CrossRefGoogle Scholar
  33. 33.
    Hauser, M.D., Chomsky, N., Fitch, W.T.: The faculty of language: What is it, who has it, and how did it evolve? Science 298, 1569–1579 (2002)CrossRefGoogle Scholar
  34. 34.
    Chomsky, N.: Three models for the description of language. IRE Transactions on information theory 2, 113–124 (1956)CrossRefGoogle Scholar
  35. 35.
    Murdock, B.B.: Serial order effects in a distributed-memory model. In: Gorfein, D.S., Hoffman, R.R. (eds.) MEMORY AND LEARNING: The Ebbinghaus Centennial Conference, pp. 277–310. Lawrence Erlbaum Associates, Mahwah (1987)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Simon D. Levy
    • 1
  • Simon Kirby
    • 2
  1. 1.Computer Science DepartmentWashington and Lee UniversityLexingtonUSA
  2. 2.Language Evolution and Computation Research Unit, School of Philosophy, Psychology and Language SciencesUniversity of EdinburghEdinburghUK

Personalised recommendations