Advertisement

An Incremental Approach to Language Acquisition: Thematic Role Assignment with Echo State Networks

  • Xavier Hinaut
  • Stefan Wermter
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8681)

Abstract

In previous research a model for thematic role assignment (θRARes) was proposed, using the Reservoir Computing paradigm. This language comprehension model consisted of a recurrent neural network (RNN) with fixed random connections which models distributed processing in the prefrontal cortex, and an output layer which models the striatum. In contrast to this previous batch learning method, in this paper we explored a more biological learning mechanism. A new version of the model (i-θRARes) was developed that permitted incremental learning, at each time step. Learning was based on a stochastic gradient descent method. We report here results showing that this incremental version was successfully able to learn a corpus of complex grammatical constructions, reinforcing the neurocognitive plausibility of the model from a language acquisition perspective.

Keywords

reservoir computing recurrent neural network language acquisition incremental learning anytime processing grammar acquisition 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bates, E., McNew, S., MacWhinney, B., Devescovi, A., Smith, S.: Functional constraints on sentence processing: A cross-linguistic study. Cognition 11, 245–299 (1982)CrossRefGoogle Scholar
  2. 2.
    Dominey, P.F., Voegtlin, T.: Learning word meaning and grammatical constructions from narrated video events. In: Proc HLT-NAACL (2003)Google Scholar
  3. 3.
    Dominey, P.F., Hoen, M., Inui, T.: A neurolinguistic model of grammatical construction processing. J. Cogn. Neurosci. 18, 2088–2107 (2006)CrossRefGoogle Scholar
  4. 4.
    Farkas, I., Crocker, M.W.: Syntactic systematicity in sentence processing with a recurrent self-organizing network. Neurocomputing 71, 1172–1179 (2008)CrossRefGoogle Scholar
  5. 5.
    Goldberg, A.E.: Constructions: a new theoretical approach to language. Trends. Cogn. Sci. 7, 219–224 (2003)CrossRefGoogle Scholar
  6. 6.
    Hinaut, X., Dominey, P.F.: A three-layered model of primate prefrontal cortex encodes identity and abstract categorical structure of behavioral sequences. J. Physiol. - Paris 105, 16–24 (2011)CrossRefGoogle Scholar
  7. 7.
    Hinaut, X., Dominey, P.F.: On-Line Processing of Grammatical Structure Using Reservoir Computing. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds.) ICANN 2012, Part I. LNCS, vol. 7552, pp. 596–603. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Hinaut, X., Dominey, P.F.: Real-Time Parallel Processing of Grammatical Structure in the Fronto-Striatal System: A Recurrent Network Simulation Study Using Reservoir Computing. PloS ONE 8(2), e52946 (2013)Google Scholar
  9. 9.
    Hinaut, X., Petit, M., Pointeau, G., Dominey, P.F.: Exploring the Acquisition and Production of Grammatical Constructions Through Human-Robot Interaction with Echo State Networks. Front. Neurorobot. 8, 16 (2014)CrossRefGoogle Scholar
  10. 10.
    Hoerzer, G.M., Legenstein, R., Maass, W.: Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. Cereb Cortex, Advance online publication (November 11, 2012) (retrieved)Google Scholar
  11. 11.
    Jaeger, H., Haas, H.: Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 (2004)CrossRefGoogle Scholar
  12. 12.
    Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3, 127–149 (2009)CrossRefGoogle Scholar
  13. 13.
    Miikkulainen, R.: Subsymbolic case-role analysis of sentences with embedded clauses. Cognitive Sci. 20, 47–73 (1996)CrossRefGoogle Scholar
  14. 14.
    Tong, M.H., Bickett, A.D., Christiansen, E.M., Cottrell, G.W.: Learning grammatical structure with Echo State Networks. Neural Networks 20, 424–432 (2007)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Xavier Hinaut
    • 1
  • Stefan Wermter
    • 1
  1. 1.Department Informatics, Knowledge TechnologyUniversity of HamburgHamburgGermany

Personalised recommendations