Skip to main content
SpringerLink
Log in

Learning Distributed Representations of Conceptual Knowledge and their Application to Script-based Story Processing

  • Chapter
Connectionist Natural Language Processing

Abstract

We propose a new method for developing distributed connectionist representations in order to serve as an adequate foundation for constructing and manipulating conceptual knowledge. In our approach, distributed representations of semantic relations (i.e. propositions) are formed by recirculating the hidden layer in two auto-associative recurrent PDP (parallel distributed processing) networks, and our experiments show that the resulting distributed semantic representations (DSRs) have many desirable properties such as automaticity, portability, structure-encoding ability and similarity-based distributed representations. We have constructed a symbolic/connectionist hybrid script-based story processing system dynasty (DYNAmic STory understanding sYstem) which incorporates DSR learning and 6 script related processing modules. Each module communicates through a global dictionary, where DSRs are stored, dynasty is able to (1) learn similarity-based distributed representations of concepts and events in everyday scriptal experiences, (2) perform script-based causal chain completion inferences according to the acquired sequential knowledge, and (3) perform script role association and retrieval during script application.

This research was supported in part by a contract from the JTF program of the DoD, monitored by JPL, and by an ITA Foundation Grant. The simulations were carried out on equipment awarded to the UCLA AI Laboratory by Hewlett Packard. Thanks to Risto Miikkulainen for providing a clustering code and to Trent Lange and John Reeves for proof reading a draft of this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Allen, R.B. (1988) Sequential connectionist networks for answering simple questions about a microworld. In Proceedings of the Tenth Annual Cognitive Science Society Conference, pp. 489–495. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Bower, G.H., Black, J.B. & Turner, T.J. (1979) Scripts in memory for text. Cognitive Psychology, 11, 177–220.

    Article  Google Scholar 

  • Chalmers, D.J. (1990) Syntatic transformations on distributed representations. Connection Science, 2, 53–63.

    Article  Google Scholar 

  • Chun, H.W. & Mimo, A. (1987) A model of schema selection using marker passing and connectionist spreading activation. In Proceedings of the Ninth Annual Cognitive Science Society Conference, pp. 887–896. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Cullingford, R.E. (1978) Script application: computer understanding of newspaper stories. PhD thesis, Department of Computer Science, Yale University. Technical Report 116.

    Google Scholar 

  • Dolan, C.P. (1989) Tensor manipulation networks: connectionist and symbolic approaches to comprehension, learning and planning. PhD thesis, Computer Science Department, UCLA.

    Google Scholar 

  • Dolan, C.P. & Dyer, M.G. (1987) Symbolic schemata, role binding, and the evolution of structure in connectionist memories. In Proceedings of the IEEE First Annual International Conference on Neural Networks, Vol. 2, pp. 287–298. IEEE.

    Google Scholar 

  • Dolan, C.P. & Smolensky, P. (1989) Implementing a connectionist production system using tensor products. In D. S. Touretzky, G. E. Hinton & T. J. Sejnowski (Eds) Proceedings of the 1988 Connectionist Models Summer School, pp. 265–272. Los Altos, CA: Morgan Kaufmann.

    Google Scholar 

  • Dyer, M.G. (1990) Symbolic NeuroEngineering for natural language processing: a multilevel research approach. In J. Barnden & J. Pollack (Eds) Advances in Connectionist and Neural Computation Theory. Norwood, NJ: Ablex (in press).

    Google Scholar 

  • Dyer, M.G., Cullingford, R.E. & Alvarado, S. (1987) Scripts. In S. C Shapiro (Ed.) Encyclopedia of Artificial Intelligence, pp. 980–994. New York: Wiley.

    Google Scholar 

  • Dyer, M.G., Flowers, M. & Wang, A. (1988) Weight matrix = pattern of activation: encoding semantic networks as distributed representations in DUAL, a PDP architecture. Technical Report UCLA-AI-88-5, Artificial Intelligence Laboratory, Computer Science Department, University of California, Los Angeles.

    Google Scholar 

  • Dyer, M.G., Flowers, M. & Wang, A. (1989) Distributed symbol discovery through symbol recirculation: toward natural language processing in distributed connectionist networks. In R. Reilly & N. Sharkey (Eds) Connectionist Approaches to Natural Language Understanding. Hillsdale, NJ: Erlbaum (in press).

    Google Scholar 

  • Elman, J.L. (1988) Finding structure in time. Technical Report 8801, Center for Research in Language, University of California, San Diego.

    Google Scholar 

  • Feldman, J.A. (1986) Neural representation of conceptual knowledge. Technical Report TR 189, Department of Computer Science, University of Rochester, NY.

    Google Scholar 

  • Feldman, J.A. & Ballard, D.H. (1982) Connectionist models and their properties. Cognitive Science, 6, 205–254.

    Article  Google Scholar 

  • Fillmore, C.J. (1968) The case for case. In E. Bach & R. T. Harms (Eds) Universals in Linguistic Theory, pp. 1–90. New York: Holt, Rinehart & Winston.

    Google Scholar 

  • Fodor, J. & Pylyshyn, Z. (1988) Connectionism and cognitive architecture: a critical analysis. Cognition, 28, 3–71.

    Article  Google Scholar 

  • Golden, R.M. (1986) Representing causal schemata in connectionist systems. In Proceedings of the Eighth Annual Cognitive Science Society Conference, pp. 13–22. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Hanson, S.J. & Kegl, J. (1987) Parsnip: a connectionist network that learns natural language grammar from exposure to natural language sentences. In Proceedings of the Ninth Annual Cognitive Science Society Conference, pp. 106–119. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Hartigan, J.A. (1975) Clustering Algorithms. New York: Wiley.

    Google Scholar 

  • Hinton, G.E. (1986) Learning distributed representations of concepts. In Proceedings of the Eighth Annual Cognitive Science Society Conference, pp. 2–12. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Hinton, G.E. (1988) Representing part-whole hierarchies in connectionist networks. In Proceedings of the Tenth Annual Cognitive Science Society Conference, pp. 48–54. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Hinton, G.E., McClelland, J.L. & Rumelhart, D.E. (1986) Distributed representations. In D. E. Rumelhart & J. L. McClelland (Eds) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. I, Foundations, pp. 77–109. Cambridge, MA: MIT Press.

    Google Scholar 

  • Lee, G. (1990a) Distributed semantic representations for goal/plan analysis of natural language stories in a connectionist architecture, PhD thesis, Computer Science Department, University of California, Los Angeles (in preparation).

    Google Scholar 

  • Lee, G. (1990b) DYNASTYII: a neural network model of goal-based story processing system. Unpublished research report, Computer Science Department, UCLA.

    Google Scholar 

  • Lee, G., Flowers, M. & Dyer, M.G. (1989a) Learning distributed representations of conceptual knowledge. Technical Report UCLA-AI-89-13, Artificial Intelligence Laboratory, Computer Science Department, University of California, Los Angeles.

    Google Scholar 

  • Lee, G., Flowers, M. & Dyer, M.G. (1989b) A symbolic/connectionist script applier mechanism. In Proceedings of the Eleventh Annual Cognitive Science Society Conference, pp. 714–721. Hillsdale, NJ: Cognitive Science Society, Erlbaum.

    Google Scholar 

  • McClelland, J.L. & Kawamoto, A.H. (1986) Mechanisms of sentence processing: assigning roles to constituents. In J. L. McClelland & D. E. Rumelhart (Eds) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. II, Psychological and Biological Models, pp. 272–326. Cambridge, MA: MIT Press.

    Google Scholar 

  • Mead, C. (1987) Silicon models of neural computation. In Proceedings of the IEEE First Annual International Conference on Neural Networks. IEEE.

    Google Scholar 

  • Miikkulainen, R. (1990) A distributed feature map model of the lexicon. Technical Report UCLA-AI-90-04, Artificial Intelligence Laboratory, Computer Science Department, University of California, Los Angeles.

    Google Scholar 

  • Miikkulainen, R. & Dyer, M.G. (1988) Forming global representations with extended back-propagation. In Proceedings of the IEEE Second Annual International Conference on Neural Networks, Vol. 1, pp. 285–292. IEEE.

    Article  Google Scholar 

  • Miikkulainen, R. & Dyer, M.G. (1989) A modular neural network architecture for sequential paraphrasing of script-based stories. In Proceedings of the International Joint Conference on Neural Networks, Vol. 2, pp. 49–56. IEEE.

    Article  Google Scholar 

  • Pollack, J.B. (1988) Recursive auto-associative memory: devising compositional distributed representations. Technical Report MCCS-88-124, Computing Research Laboratory, New Mexico State University.

    Google Scholar 

  • Pollack, J.B. (1990) Recursive distributed representations. Artificial Intelligence, 45, Special issue on connectionist symbol processing.

    Google Scholar 

  • Rumelhart, D.E., Hinton, G.E. & Williams, R.J. (1986a) Learning internal representations by error propagation. In D. E. Rumelhart & J. L. McClelland (Eds) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. I, Foundations, pp. 318–362. Cambridge, MA: MIT Press.

    Google Scholar 

  • Rumelhart, D.E., McClelland, J.L. & the PDP Research Group (1986b) Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Cambridge, MA: MIT Press.

    Google Scholar 

  • Rumelhart, D.E., Smolensky, P., McClelland, J.L. & Hinton, G.E. (1986c) Schemata and sequential thought processes in pdp models. In J. L. McClelland & D. E. Rumelhart (Eds) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. II, Psychological and Biological Models, pp. 7–57. Cambridge, MA: MIT Press.

    Google Scholar 

  • Schank, R. (1973) Identification of conceptualization underlying natural language. In R. Schank & R. Colby (Eds) Computer Models of Thought and Language, pp. 187–248. San Francisco, CA: Freeman.

    Google Scholar 

  • Schank, R. & Abelson, R. (1977) Scripts, Plans, Goals, and Understanding—an Inquiry into Human Knowledge Structures. The Artificial Intelligence Series. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Schank, R. & Riesbeck, C.K. (Eds) (1981) Inside Computer Understanding. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Smolensky, P. (1987a) A method for connectionist variable binding. Technical Report CU-CS-356-87, Department of Computer Science and Institute of Cognitive Science, University of Colorado, Boulder.

    Google Scholar 

  • Smolensky, P. (1987b) On variable binding and the representation of symbolic structures in connectionist systems. Technical Report CU-CS-355-87, Department of Computer Science and Institute of Cognitive Science, University of Colorado, Boulder.

    Google Scholar 

  • Smolensky, P. (1988) On the proper treatment of connectionism. Behavioral and Brain Sciences, 11, 1–74.

    Article  Google Scholar 

  • St John, M.F. & McClelland, J.L. (1989) Applying contextual constraints in sentence comprehension. In D. S. Touretzky, G. E. Hinton & T. J. Sejnowski (Eds) Proceedings of the 1988 Connectionist Models Summer School, pp. 338–346. Los Altos, CA: Morgan Kaufmann.

    Google Scholar 

  • Touretzky, D.S. & Geva, S.A. (1988) A distributed connectionist representation for concept structures. In Proceedings of the Tenth Annual Cognitive Science Society Conference, pp. 155–163. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Touretzky, D.S. & Hinton, G.E. (1988) A distributed connectionist production system. Cognitive Science, 12, 423–436.

    Article  Google Scholar 

  • Waltz, D.L. & Pollack, J.B. (1985) Massively parallel parsing: a strongly interactive model of natural language interpretation. Cognitive Science, 9, 51–74.

    Article  Google Scholar 

  • Wilensky, R. (1978) Understanding goal-based stories. PhD thesis, Computer Science Department, Yale University.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Artificial Intelligence Laboratory, Computer Science Department, University of California, 3532 Boelter Hall, Los Angeles, CA, 90024, USA

    Geunbae Lee, Margot Flowers & Michael G. Dyer

Authors
  1. Geunbae Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Margot Flowers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael G. Dyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Exeter, UK

    Noel Sharkey

Rights and permissions

Reprints and permissions

Copyright information

© 1992 Springer Science+Business Media Dordrecht

About this chapter

Cite this chapter

Lee, G., Flowers, M., Dyer, M.G. (1992). Learning Distributed Representations of Conceptual Knowledge and their Application to Script-based Story Processing. In: Sharkey, N. (eds) Connectionist Natural Language Processing. Springer, Dordrecht. https://doi.org/10.1007/978-94-011-2624-3_11

Download citation

  • DOI: https://doi.org/10.1007/978-94-011-2624-3_11

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-010-5160-6

  • Online ISBN: 978-94-011-2624-3

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation