Advertisement

Measure for Measure: Towards Increased Component Comparability and Exchange

  • Stephan Oepen
  • Ulrich Callmeier
Part of the Text, Speech and Language Technology book series (TLTB, volume 23)

Abstract

Over the past few years, significant progress has been made in efficient processing with wide-coverage HPSG grammars. HPSG-based parsing systems are now available that can process medium-complexity sentences (of ten to twenty words, say) in average parse times equivalent to real (i.e. human reading) time. A large number of engineering improvements in current HPSG systems have been achieved through collaboration of multiple research centers and mutual exchange of experience, encoding techniques, algorithms, and even pieces of software. This article presents an approach to grammar and system engineering, termed competence & performance profiling, that makes systematic experimentation and the precise empirical study of system properties a focal point in development. Adapting the profiling metaphor familiar from software engineering to constraint-based grammars and parsers enables developers to maintain an accurate record of system evolution, identify grammar and system deficiencies quickly, and compare to earlier versions or between different systems. We discuss a number of example problems that motivate the experimental approach, and apply the empirical methodology in a fairly detailed discussion of progress made during a development period of three years.

Keywords

Memory Consumption Performance Profile Computational Linguistics Active Edge Argument Position 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aït-Kaci, H. (1991). Warren’s Abstract Machine: A Tutorial Reconstruction. MIT Press, Cambridge, MA.Google Scholar
  2. Bouma, G. and van Noord, G. (1993). Head-driven parsing for lexicalist grammars. Experimental results. In Proceedings of the 6th Conference of the European Chapter of the ACL, pages 71–80, Utrecht, The Netherlands.Google Scholar
  3. Callmeier, U. (2000). PET — A platform for experimentation with efficient HPSG processing techniques. Natural Language Engineering, 6(1) (Special Issue on Efficient Processing with HPSG):99–108.CrossRefGoogle Scholar
  4. Calder, J. (1993). Graphical interaction with constraint-based grammars. In Proceedings of the 3rd Pacific Rim Conference on Computational Linguistics, pages 160–169, Vancouver, BC, Canada.Google Scholar
  5. Callmeier, U. (2002). Preprocessing and encoding techniques in PET. In Oepen, S., Flickinger, D., Tsujii, J., and Uszkoreit, H., editors, Collaborative Language Engineering. A Case Study in Efficient Grammar-based Processing. CSLI Publications, Stanford, CA.Google Scholar
  6. Carpenter, B. (1992). The Logic of Typed Feature Structures. Cambridge University Press, Cambridge, UK.zbMATHCrossRefGoogle Scholar
  7. Carroll, J. (1994). Relating complexity to practical performance in parsing with wide-coverage unification grammars. In Proceedings of the 32nd Meeting of the Association for Computational Linguistics, pages 287–294, Las Cruces, NM.Google Scholar
  8. Carroll, J., Copestake, A., Flickinger, D., and Poznanski, V. (1999). An efficient chart generator for (semi-)lexicalist grammars. In Proceedings of the 7th European Workshop on Natural Language Generation, pages 86–95, Toulouse, France.Google Scholar
  9. Ciortuz, L. (2001). LIGHT. A feature constraint language applied to parsing with large-scale HPSG grammars. Unpublished DFKI research report, Deutsches Forschungszentrum für Künstliche Intelligenz, Saarbrücken, Germany.Google Scholar
  10. Copestake, A. (1992). The ACQUILEX LKB. Representation issues in semiautomatic acquisition of large lexicons. In Proceedings of the 3rd ACL Conference on Applied Natural Language Processing, pages 88–96, Trento, Italy.Google Scholar
  11. Erbach, G. (1991a). An environment for experimenting with parsing strategies. In Mylopoulos, J. and Reiter, R., editors, Proceedings of the 12th International Joint Conference on Artificial Intelligence, pages 931–937, San Mateo, CA. Morgan Kaufmann Publishers.Google Scholar
  12. Erbach, G. (1991b). A flexible parser for a linguistic development environment. In Herzog, O. and Rollinger, C.-R., editors, Text Understanding in LILOG, pages 74–87. Springer, Berlin, Germany.CrossRefGoogle Scholar
  13. Flickinger, D. (2000). On building a more efficient grammar by exploiting types. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG): 15–28.CrossRefGoogle Scholar
  14. Flickinger, D., Nerbonne, J., Sag, I. A., and Wasow, T. (1987). Toward evaluation of NLP systems. Technical report, Hewlett-Packard Laboratories. Distributed at the 24th Annual Meeting of the Association for Computational Linguistics.Google Scholar
  15. Flickinger, D. P. and Sag, I. A. (1998). Linguistic Grammars Online. A multipurpose broad-coverage computational grammar of English. In CSLI Bulletin 1999, pages 64–68, Stanford, CA. CSLI Publications.Google Scholar
  16. Gerdemann, D. (1995). Term encoding of typed feature structures. In Proceedings of the 4th International Workshop on Parsing Technologies, pages 89–97, Prague, Czech Republic.Google Scholar
  17. Götz, T. (1993). A normal form for typed feature structures. Magisterarbeit, Universität Tübingen, Tübingen, Germany.Google Scholar
  18. Kay, M. (1989). Head-driven parsing. In Proceedings of the 1st International Workshop on Parsing Technologies, pages 52–62, Pittsburgh, PA.Google Scholar
  19. Kiefer, B., Krieger, H.-U., Carroll, J., and Malouf, R. (1999). A bag of useful techniques for efficient and robust parsing. In Proceedings of the 37th Meeting of the Association for Computational Linguistics, pages 473–480, College Park, MD.Google Scholar
  20. Krieger, H.-U. and Schäfer, U. (1994). TDL — A type description language for constraint-based grammars. In Proceedings of the 15th International Conference on Computational Linguistics, pages 893–899, Kyoto, Japan.Google Scholar
  21. Malouf, R., Carroll, J., and Copestake, A. (2000). Efficient feature structure operations without compilation. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG):29–46.CrossRefGoogle Scholar
  22. van Noord, G. (1997). An efficient implementation of the head-corner parser. Computational Linguistics, 23 (3):425–456.Google Scholar
  23. van Noord, G. and Bouma, G. (1997). HDRUG: A flexible and extendible development environment for natural language processing. In Proceedings of the EACL/ACL workshop ENVGRAM, Madrid, Spain.Google Scholar
  24. Oepen, S. and Carroll, J. (2000). Performance profiling for parser engineering. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG):81–97.CrossRefGoogle Scholar
  25. Oepen, S., Flickinger, D., Tsujii, J., and Uszkoreit, H., editors (2002). Collaborative Language Engineering. A Case Study in Efficient Grammar-based Processing. CSLI Publications, Stanford, CA.Google Scholar
  26. Oepen, S. and Flickinger, D. P. (1998). Towards systematic grammar profiling. Test suite technology ten years after. Journal of Computer Speech and Language, 12 (4) (Special Issue on Evaluation):411–436.CrossRefGoogle Scholar
  27. Oepen, S., Netter, K., and Klein, J. (1997). TSNLP — Test Suites for Natural Language Processing. In Nerbonne, J., editor, Linguistic Databases, pages 13–36. CSLI Publications, Stanford, CA.Google Scholar
  28. Shakespeare, W. (1623). Measure for Measure. I. Iaggard and E. Blount, London, UK, first folio edition.Google Scholar
  29. Siegel, M. (2000). HPSG analysis of Japanese. In Wahlster, W., editor, Verbmobil. Foundations of Speech-to-Speech Translation, pages 265–280. Springer, Berlin, Germany.Google Scholar
  30. Tomabechi, H. (1991). Quasi-destructive graph unification. In Proceedings of the 29th Meeting of the Association for Computational Linguistics, pages 315–322, Berkeley, CA.Google Scholar
  31. Torisawa, K. and Tsujii, J. (1996). Computing phrasal signs in HPSG prior to parsing. In Proceedings of the 16th International Conference on Computational Linguistics, pages 949–955, Copenhagen, Denmark.Google Scholar
  32. Uszkoreit, H., Backofen, R., Busemann, S., Diagne, A. K., Hinkelman, E. A., Kasper, W., Kiefer, B., Krieger, H.-U., Netter, K., Neumann, G., Oepen, S., and Spackman, S. P. (1994). DISCO — an HPSG-based NLP system and its application for appointment scheduling. In Proceedings of the 15th International Conference on Computational Linguistics, Kyoto, Japan.Google Scholar
  33. Wahlster, W., editor (2000). Verbmobil. Foundations of Speech-to-Speech Translation. Springer, Berlin, Germany.zbMATHGoogle Scholar
  34. Wroblewski, D. A. (1987). Non-destructive graph unification. In Proceedings of the 6th National Conference on Artificial Intelligence, pages 582–587, Seattle, WA.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Stephan Oepen
    • 1
  • Ulrich Callmeier
    • 2
  1. 1.Center for the Study of Language and InformationStanford UniversityStanford(USA)
  2. 2.Department of Computational LinguisticsSaarland UniversitySaarbrücken(Germany)

Personalised recommendations