Skip to main content

Evaluating Language Technologies: The MULTIDOC Approach to Taming the Knowledge Soup

  • Conference paper
  • First Online:

Part of the Lecture Notes in Computer Science book series (LNAI,volume 1529)


In this paper we report on ongoing verification and validation work within the MULTIDOC project. This project is situated in the field of multilingual automotive product documentation. One central task is the evaluation of existing off-the-shelf and research based language technology (LT) products and components for the purpose of supporting or even reorganising the documentation production chain along three diagnostic dimensions: the process proper, the documentation quality and the translatability of the process output. In this application scenario, LT components shall control and ensure that predefined quality criteria are applicable and measurable to the documentation end-product as well as to the information objects that form the basic building blocks of the end-product. In this scenario, multilinguality is of crucial importance. It shall be introduced or prepared, and maintained as early as possible in the documentation workflow to ensure a better and faster translation process. A prerequisite for the evaluation process is the thorough definition of these dimensions in terms of user quality requirements and LT developer quality requirements. In our approach, we define the output quality of the whole documentation process as the pivot where user requirements and developer requirements shall meet. For this, it turned out that a so-called “braided” diagnostic evaluation is very well suited to cover both views. Since no generally approved standards or even valid specifications for standards exist for the evaluation of LT products, we have adjusted existing standards for the evaluation of software products, in particular ISO 9001, ISO 9000-3, ISO/IEC 12119, ISO 9004 and ISO 9126. This is feasible because an LT product consists of a software part and a lingware part. The adaptation had to be accomplished for the latter part.


  • Machine Translation
  • Quality Requirement
  • Information Object
  • Repair Operation
  • Language Technology

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/3-540-49478-2_22
  • Chapter length: 14 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   74.99
Price excludes VAT (USA)
  • ISBN: 978-3-540-49478-2
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.00
Price excludes VAT (USA)


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. EAGLES: Evaluation of Natural Language Processing System. Final Report, EAGLES Document EAG-EWG-PR.2, Geneva, Switzerland (1995)

    Google Scholar 

  2. Haller, J.: MULTILINT-Multilingual Documentation with Linguistic Intelligence. In: Proceedings of ‘Translating and the Computer’, ASLIB, London, Great Britain (1996)

    Google Scholar 

  3. Haller, J. and Schütz, J.: Integration linguistischer Intelligenz in die multilinguale technische Dokumentation. In Proceedings of EUROMAP Forum’ sprache ohne Grenzen’, M:unchen, Germany (1997)

    Google Scholar 

  4. Maas, H.D.: Multilinguale Textverarbeitung mit MPRO. In: Lobin, G., Lohse, H. Piotrowski, S and Poláková, E. (Eds.): Europäische Kommunikationskybernetik heute und morgen, KoPäd, München, Germany (1998) 167–173

    Google Scholar 

  5. Nübel, R.: End-to-End Evaluation in Verbmobil I. In: Proceedings of Machine Translation Summit VI, San Diego, California, USA (1997) 232–239

    Google Scholar 

  6. OpenTag-Formal Specifications. Version 1.1b April-22-1998, Last edit: May-01-1998. Available on the Web at (1998)

  7. Schütz, J.: Language Engineering-Fixing Positions. IAI Memo 0695, Saarbrücken, Germany. Available on the Web at (1995)

  8. Schütz, J.: Combining Language Technology and Web Technology to Streamline an Automotive Hotline Support Service. In: Proceedings of AMTA 96, Montreal, Canada (1996) 180–189

    Google Scholar 

  9. Schütz, J.: Utilizing Evaluation in Networked Machine Translation. In: Proceedings of the 7th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI) 1997, Santa Fe, New Mexico, USA (1997) 208–215

    Google Scholar 

  10. Schütz, J. and Nübel, R.: Multi-purpose vs. Specific Application: Diagnostic Evaluation of Multilingual Language Technologies. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain (1998) 1251–1254

    Google Scholar 

  11. Thurmair, G.: Exchange Interfaces for Translation Tools. In Proceedings of MT Summit VI, San Diego, California, USA (1997) 74–92

    Google Scholar 

  12. Thurmair, G., Ritzke, J. and McCormik, S.: The Open Lexicon Interchange Format-OLIF. OTELO Report available on the Web at (1998)

  13. TMX Format Specifications. Version 1.0 November-25-1997. Available on the Web at (1997)

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 1998 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Schütz, J., Nübel, R. (1998). Evaluating Language Technologies: The MULTIDOC Approach to Taming the Knowledge Soup. In: Farwell, D., Gerber, L., Hovy, E. (eds) Machine Translation and the Information Soup. AMTA 1998. Lecture Notes in Computer Science(), vol 1529. Springer, Berlin, Heidelberg.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-65259-5

  • Online ISBN: 978-3-540-49478-2

  • eBook Packages: Springer Book Archive

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.