Applied Intelligence

, Volume 19, Issue 1–2, pp 83–99 | Cite as

Towards Effective Parsing with Neural Networks: Inherent Generalisations and Bounded Resource Effects

  • Peter C.R. Lane
  • James B. Henderson


This article explores how the effectiveness of learning to parse with neural networks can be improved by including two architectural features relevant to language: generalisations across syntactic constituents and bounded resource effects. A number of neural network parsers have recently been proposed, each with a different approach to the representational problem of outputting parse trees. In addition, some of the parsers have explicitly attempted to capture an important regularity within language, which is to generalise information across syntactic constituents. A further property of language is that natural bounds exist for the number of constituents which a parser need retain for later processing. Both the generalisations and the resource bounds may be captured in architectural features which enhance the effectiveness and efficiency of learning to parse with neural networks. We describe a number of different types of neural network parser, and compare them with respect to these two features. These features are both explicitly present in the Simple Synchrony Network parser, and we explore and illustrate their impact on the process of learning to parse in some experiments with a recursive grammar.

neural networks resource effects structured representations syntactic parsing systematicity 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    E. Charniak, Statistical Language Learning, MIT Press: Cambridge, MA, 1993.Google Scholar
  2. 2.
    M. Johnson, “PCFG models of linguistic tree representations,” Computational Linguistics, vol. 24, pp. 613–632, 1998.Google Scholar
  3. 3.
    R.F. Hadley and M.B. Hayward, “Strong semantic systematicity from Hebbian connectionist learning,” Minds and Machines, vol. 7, pp. 1–37, 1997.Google Scholar
  4. 4.
    J.B. Henderson, “A neural network parser that handles sparse data,” in Proceedings of the 6th International Workshop on Parsing Technologies, Trento, Italy, 2000, pp. 123–134.Google Scholar
  5. 5.
    J.B. Henderson and P.C.R. Lane, “A connectionist architecture for learning to parse,” in Proceedings of the 17th International Conference on Computational Linguistics and the 36 th Annual Meeting of the Association for Computational Linguistics (COLING-ACL’98), University of Montreal, Canada, 1998, pp. 531–537.Google Scholar
  6. 6.
    E.K.S. Ho and L.W. Chan, “How to design a connectionist holistic parser,” Neural Computation, vol. 11, no.8, pp. 1995–2016, 1999.Google Scholar
  7. 7.
    P.C.R. Lane, “Simple Synchrony Networks: A new connectionist architecture applied to natural language parsing,” Ph.D. Thesis, Department of Computer Science, University of Exeter, England, 2000.Google Scholar
  8. 8.
    P.C.R. Lane and J.B. Henderson, “Simple Synchrony Networks: Learning to parse natural language with Temporal Synchrony Variable Binding,” in Proceedings of the Eighth International Conference on Artificial Neural Networks, edited by L. Niklasson, M. Boden, and T. Ziemke, Skövde, Sweden, 1998, pp. 615–620.Google Scholar
  9. 9.
    P.C.R. Lane and J.B. Henderson, “Incremental syntactic parsing of natural language corpora with Simple Synchrony Networks,” IEEE Transactions on Knowledge and Data Engineering, vol. 13, pp. 219–231, 2001.Google Scholar
  10. 10.
    R. Miikkulainen, “Subsymbolic case-role analysis of sentences with embedded clauses,” Cognitive Science, vol. 20, pp. 47–73, 1996.Google Scholar
  11. 11.
    J.A. Fodor and Z.W. Pylyshyn, “Connectionism and cognitive architecture: A critical analysis,” Cognition, vol. 28, pp. 3–71, 1988.Google Scholar
  12. 12.
    N. Cowan, “The magical number 4 in short-term memory: A reconsideration of mental storage capacity,” Behavioral and Brain Sciences, vol. 24, no.1, pp. 87–185, 2001.Google Scholar
  13. 13.
    G.A. Miller, “The magical number seven, plus or minus two: Some limits on our capacity for processing information,” Psychological Review, vol. 63, pp. 81–97, 1956.Google Scholar
  14. 14.
    N. Chomsky, Aspects of the Theory of Syntax, MIT Press: Cambridge, MA, 1965.Google Scholar
  15. 15.
    J. Kimball, “Seven principles of surface structure parsing in natural language,” Cognition, vol. 2, pp. 15–47, 1976.Google Scholar
  16. 16.
    M. Marcus, A Theory of Syntactic Recognition for Natural Language, MIT Press: Cambridge, MA, 1980.Google Scholar
  17. 17.
    D. Chalmers, “Syntactic transformations of distributed representations,” in Connectionist Natural Language Processing, edited by N. Sharkey, Kluwer: Boston, 1992, pp. 46–55.Google Scholar
  18. 18.
    L. Chrisman, “Learning recursive distributed representations for holistic computation,” Connection Science, vol. 3, pp. 345–366, 1991.Google Scholar
  19. 19.
    R.F. Hadley, “Systematicity in connectionist language learning,” Mind and Language, vol. 9, pp. 247–272, 1994.Google Scholar
  20. 20.
    R.F. Hadley, “Systematicity revisited: Reply to Christiansen and Chater and Niklasson and van Gelder,” Mind and Language, vol. 9, pp. 431–444, 1994.Google Scholar
  21. 21.
    K. Aizawa, “Exhibiting versus explaining systematicity: A reply to Hadley and Hayward,” Minds and Machines, vol. 7, pp. 39–55, 1997.Google Scholar
  22. 22.
    L. Shastri and V. Ajjanagadde, “From simple associations to systematic reasoning: A connectionist representation of rules, variables, and dynamic bindings using temporal synchrony,” Behavioral and Brain Sciences, vol. 16, pp. 417–494, 1993.Google Scholar
  23. 23.
    J.B. Henderson, “A connectionist architecture with inherent systematicity,” in Proceedings of the Eighteenth Conference of the Cognitive Science Society, La Jolla, CA, 1996, pp. 574–579.Google Scholar
  24. 24.
    E. Gibson, “A computational theory of human linguistic processing: Memory limitations and processing breakdown,” Ph.D. Thesis, Carnegie Mellon University, Pittsburgh, PA, 1991.Google Scholar
  25. 25.
    K. Hornik, W. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, pp. 359–366, 1989.Google Scholar
  26. 26.
    E.K.S. Ho and L.W. Chan, “Confluent preorder parsing of deterministic grammars,” Connection Science, vol. 9, pp. 269–293, 1997.Google Scholar
  27. 27.
    R. Callan and D. Palmer-Brown, “An analytical technique for fast and reliable derivation of connectionist symbol structure representations,” Connection Science, vol. 9, no.2, pp. 139–159, 1997.Google Scholar
  28. 28.
    J. Pollack, “Recursive distributed representations,” Artificial Intelligence, vol. 46, pp. 77–105, 1990.Google Scholar
  29. 29.
    D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, edited by D.E. Rumelhart, J.L. McClelland, and the PDP Research Group, MIT Press: Cambridge, MA, vol. 1, 1986.Google Scholar
  30. 30.
    J.B. Henderson, “Description based parsing in a connectionist network,” Ph.D. Thesis, University of Pennsylvania, 1994.Google Scholar
  31. 31.
    J.L. Elman, “Distributed representations, simple recurrent networks, and grammatical structure,” Machine Learning, vol. 7, pp. 195–225, 1991.Google Scholar
  32. 32.
    R. Miikkulainen, “Natural language processing with subsymbolic neural networks,” in Neural Network Perspectives on Cognition and Adaptive Robotics, edited by A. Browne, 1997, pp. 120–139.Google Scholar
  33. 33.
    P.C.R. Lane, “Simple Synchrony Networks: Learning generalisations across syntactic constituents,” in Proceedings of the Thirteenth European Conference in Artificial Int elligence, Brighton: John Wiley & Sons, UK, 1998, pp. 469–470.Google Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Peter C.R. Lane
    • 1
  • James B. Henderson
    • 2
  1. 1.Department of Computer ScienceUniversity of Hertfordshire, Hatfield CampusHATFIELDUK
  2. 2.Department of Computer ScienceUniversity of GenevaGenève 4Switzerland

Personalised recommendations