Skip to main content

Evaluating Language Technologies: The MULTIDOC Approach to Taming the Knowledge Soup

  • Conference paper
  • First Online:
Machine Translation and the Information Soup (AMTA 1998)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 1529))

Included in the following conference series:


In this paper we report on ongoing verification and validation work within the MULTIDOC project. This project is situated in the field of multilingual automotive product documentation. One central task is the evaluation of existing off-the-shelf and research based language technology (LT) products and components for the purpose of supporting or even reorganising the documentation production chain along three diagnostic dimensions: the process proper, the documentation quality and the translatability of the process output. In this application scenario, LT components shall control and ensure that predefined quality criteria are applicable and measurable to the documentation end-product as well as to the information objects that form the basic building blocks of the end-product. In this scenario, multilinguality is of crucial importance. It shall be introduced or prepared, and maintained as early as possible in the documentation workflow to ensure a better and faster translation process. A prerequisite for the evaluation process is the thorough definition of these dimensions in terms of user quality requirements and LT developer quality requirements. In our approach, we define the output quality of the whole documentation process as the pivot where user requirements and developer requirements shall meet. For this, it turned out that a so-called “braided” diagnostic evaluation is very well suited to cover both views. Since no generally approved standards or even valid specifications for standards exist for the evaluation of LT products, we have adjusted existing standards for the evaluation of software products, in particular ISO 9001, ISO 9000-3, ISO/IEC 12119, ISO 9004 and ISO 9126. This is feasible because an LT product consists of a software part and a lingware part. The adaptation had to be accomplished for the latter part.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others


  1. EAGLES: Evaluation of Natural Language Processing System. Final Report, EAGLES Document EAG-EWG-PR.2, Geneva, Switzerland (1995)

    Google Scholar 

  2. Haller, J.: MULTILINT-Multilingual Documentation with Linguistic Intelligence. In: Proceedings of ‘Translating and the Computer’, ASLIB, London, Great Britain (1996)

    Google Scholar 

  3. Haller, J. and Schütz, J.: Integration linguistischer Intelligenz in die multilinguale technische Dokumentation. In Proceedings of EUROMAP Forum’ sprache ohne Grenzen’, M:unchen, Germany (1997)

    Google Scholar 

  4. Maas, H.D.: Multilinguale Textverarbeitung mit MPRO. In: Lobin, G., Lohse, H. Piotrowski, S and Poláková, E. (Eds.): Europäische Kommunikationskybernetik heute und morgen, KoPäd, München, Germany (1998) 167–173

    Google Scholar 

  5. Nübel, R.: End-to-End Evaluation in Verbmobil I. In: Proceedings of Machine Translation Summit VI, San Diego, California, USA (1997) 232–239

    Google Scholar 

  6. OpenTag-Formal Specifications. Version 1.1b April-22-1998, Last edit: May-01-1998. Available on the Web at (1998)

  7. Schütz, J.: Language Engineering-Fixing Positions. IAI Memo 0695, Saarbrücken, Germany. Available on the Web at (1995)

  8. Schütz, J.: Combining Language Technology and Web Technology to Streamline an Automotive Hotline Support Service. In: Proceedings of AMTA 96, Montreal, Canada (1996) 180–189

    Google Scholar 

  9. Schütz, J.: Utilizing Evaluation in Networked Machine Translation. In: Proceedings of the 7th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI) 1997, Santa Fe, New Mexico, USA (1997) 208–215

    Google Scholar 

  10. Schütz, J. and Nübel, R.: Multi-purpose vs. Specific Application: Diagnostic Evaluation of Multilingual Language Technologies. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain (1998) 1251–1254

    Google Scholar 

  11. Thurmair, G.: Exchange Interfaces for Translation Tools. In Proceedings of MT Summit VI, San Diego, California, USA (1997) 74–92

    Google Scholar 

  12. Thurmair, G., Ritzke, J. and McCormik, S.: The Open Lexicon Interchange Format-OLIF. OTELO Report available on the Web at (1998)

  13. TMX Format Specifications. Version 1.0 November-25-1997. Available on the Web at (1997)

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 1998 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Schütz, J., Nübel, R. (1998). Evaluating Language Technologies: The MULTIDOC Approach to Taming the Knowledge Soup. In: Farwell, D., Gerber, L., Hovy, E. (eds) Machine Translation and the Information Soup. AMTA 1998. Lecture Notes in Computer Science(), vol 1529. Springer, Berlin, Heidelberg.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-65259-5

  • Online ISBN: 978-3-540-49478-2

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics