Skip to main content

Systematically Grounding Language through Vision in a Deep, Recurrent Neural Network

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNAI,volume 6830)

Abstract

Human intelligence consists largely of the ability to recognize and exploit structural systematicity in the world, relating our senses simultaneously to each other and to our cognitive state. Language abilities, in particular, require a learned mapping between the linguistic input and one’s internal model of the real world. In order to demonstrate that connectionist methods excel at this task, we teach a deep, recurrent neural network—a variant of the long short-term memory (LSTM)—to ground language in a micro-world. The network integrates two inputs—a visual scene and an auditory sentence—to produce the meaning of the sentence in the context of the scene. Crucially, the network exhibits strong systematicity, recovering appropriate meanings even for novel objects and descriptions. With its ability to exploit systematic structure across modalities, this network fulfills an important prerequisite of general machine intelligence.

Keywords

  • deep recurrent neural network
  • grounded language learning
  • systematicity
  • long short-term memory

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-22887-2_12
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-642-22887-2
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Goldberg, A.E.: Learning Linguistic Patterns. The Psychology of Learning and Motivation 47, 33–63 (2006)

    CrossRef  Google Scholar 

  2. Turing, A.M.: Computing Machinery and Intelligence. Mind 59(236), 433–460 (1950)

    MathSciNet  CrossRef  Google Scholar 

  3. Searle, J.R.: Minds, Brains, and Programs. Behavioral and Brain Sciences 3(03), 417–457 (1980)

    CrossRef  Google Scholar 

  4. Harnad, S.: The Symbol Grounding Problem. Physica D: Nonlinear Phenomena 42(1-3), 335–346 (1990)

    CrossRef  Google Scholar 

  5. Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Computation 9(8), 1735–1780 (1997)

    CrossRef  Google Scholar 

  6. Gers, F.A., Cummins, F.: Learning to Forget: Continual Prediction with LSTM. Neural Computation 12(10), 2451–2471 (2000)

    CrossRef  Google Scholar 

  7. Hadley, R.: Systematicity in Connectionist Language Learning. Mind & Language (1994)

    Google Scholar 

  8. Feldman, J.A., Lakoff, G., Stolcke, A., Weber, S.H.: Miniature Language Acquisition: A Touchstone for Cognitive Science. In: Proceedings of the Annual Conference of the Cognitive Science Society, pp. 686–693 (1990)

    Google Scholar 

  9. Hadley, R.F., Cardei, V.C.: Language Acquisition from Sparse Input Without Error Feedback. Neural Networks 12(2), 217–235 (1999)

    CrossRef  Google Scholar 

  10. Riga, T., Cangelosi, A., Greco, A.: Symbol Grounding Transfer with Hybrid Self-Organizing/Supervised Neural Networks. In: Proceedings of the International Joint Conference on Neural Networks, pp. 2865–2869 (2004)

    Google Scholar 

  11. Williams, P., Miikkulainen, R.: Grounding Language in Descriptions of Scenes. In: Proceedings of the 8th Annual Conference of the Cognitive Science Society, pp. 2381–2386 (2006)

    Google Scholar 

  12. Kohonen, T.: The Self-Organizing Map. Proceedings of the IEEE 78, 1464–1480 (1990)

    CrossRef  Google Scholar 

  13. Frank, S.L., Haselager, W.F.G., van Rooij, I.: Connectionist Semantic Systematicity. Cognition 110(3), 358–379 (2009)

    CrossRef  Google Scholar 

  14. Elman, J.L.: Finding Structure in Time. Cognitive Science 14, 179–211 (1990)

    CrossRef  Google Scholar 

  15. Weems, S.A., Reggia, J.A.: Simulating Single Word Processing in the Classic Aphasia Syndromes Based on the Wernicke-Lichtheim-Geschwind Theory. Brain and Language 98(3), 291–309 (2006)

    CrossRef  Google Scholar 

  16. Tomasello, M.: Constructing a Language: A Usage-Based Theory of Language Acquisition. Harvard University Press, Cambridge (2003)

    Google Scholar 

  17. Monner, D.D., Reggia, J.A.: A Generalized LSTM-Like Training Algorithm for Second-Order Recurrent Neural Networks (2011) (submitted)

    Google Scholar 

  18. Xie, X., Seung, H.S.: Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network. Neural Computation 15(2), 441–454 (2003)

    MATH  CrossRef  Google Scholar 

  19. O’Reilly, R.C.: Generalization in Interactive Networks: The Benefits of Inhibitory Competition and Hebbian Learning. Neural Computation 13(6), 1199–1241 (2001)

    MATH  CrossRef  Google Scholar 

  20. O’Reilly, R.C., Frank, M.J.: Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia. Neural Computation 18(2), 283–328 (2006)

    MathSciNet  MATH  CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Monner, D.D., Reggia, J.A. (2011). Systematically Grounding Language through Vision in a Deep, Recurrent Neural Network. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds) Artificial General Intelligence. AGI 2011. Lecture Notes in Computer Science(), vol 6830. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22887-2_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-22887-2_12

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-22886-5

  • Online ISBN: 978-3-642-22887-2

  • eBook Packages: Computer ScienceComputer Science (R0)