Evaluating Language Technologies: The MULTIDOC Approach to Taming the Knowledge Soup

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1529)


In this paper we report on ongoing verification and validation work within the MULTIDOC project. This project is situated in the field of multilingual automotive product documentation. One central task is the evaluation of existing off-the-shelf and research based language technology (LT) products and components for the purpose of supporting or even reorganising the documentation production chain along three diagnostic dimensions: the process proper, the documentation quality and the translatability of the process output. In this application scenario, LT components shall control and ensure that predefined quality criteria are applicable and measurable to the documentation end-product as well as to the information objects that form the basic building blocks of the end-product. In this scenario, multilinguality is of crucial importance. It shall be introduced or prepared, and maintained as early as possible in the documentation workflow to ensure a better and faster translation process. A prerequisite for the evaluation process is the thorough definition of these dimensions in terms of user quality requirements and LT developer quality requirements. In our approach, we define the output quality of the whole documentation process as the pivot where user requirements and developer requirements shall meet. For this, it turned out that a so-called “braided” diagnostic evaluation is very well suited to cover both views. Since no generally approved standards or even valid specifications for standards exist for the evaluation of LT products, we have adjusted existing standards for the evaluation of software products, in particular ISO 9001, ISO 9000-3, ISO/IEC 12119, ISO 9004 and ISO 9126. This is feasible because an LT product consists of a software part and a lingware part. The adaptation had to be accomplished for the latter part.


Machine Translation Quality Requirement Information Object Repair Operation Language Technology 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    EAGLES: Evaluation of Natural Language Processing System. Final Report, EAGLES Document EAG-EWG-PR.2, Geneva, Switzerland (1995)Google Scholar
  2. 2.
    Haller, J.: MULTILINT-Multilingual Documentation with Linguistic Intelligence. In: Proceedings of ‘Translating and the Computer’, ASLIB, London, Great Britain (1996)Google Scholar
  3. 3.
    Haller, J. and Schütz, J.: Integration linguistischer Intelligenz in die multilinguale technische Dokumentation. In Proceedings of EUROMAP Forum’ sprache ohne Grenzen’, M:unchen, Germany (1997)Google Scholar
  4. 4.
    Maas, H.D.: Multilinguale Textverarbeitung mit MPRO. In: Lobin, G., Lohse, H. Piotrowski, S and Poláková, E. (Eds.): Europäische Kommunikationskybernetik heute und morgen, KoPäd, München, Germany (1998) 167–173Google Scholar
  5. 5.
    Nübel, R.: End-to-End Evaluation in Verbmobil I. In: Proceedings of Machine Translation Summit VI, San Diego, California, USA (1997) 232–239Google Scholar
  6. 6.
    OpenTag-Formal Specifications. Version 1.1b April-22-1998, Last edit: May-01-1998. Available on the Web at (1998)
  7. 7.
    Schütz, J.: Language Engineering-Fixing Positions. IAI Memo 0695, Saarbrücken, Germany. Available on the Web at (1995)
  8. 8.
    Schütz, J.: Combining Language Technology and Web Technology to Streamline an Automotive Hotline Support Service. In: Proceedings of AMTA 96, Montreal, Canada (1996) 180–189Google Scholar
  9. 9.
    Schütz, J.: Utilizing Evaluation in Networked Machine Translation. In: Proceedings of the 7th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI) 1997, Santa Fe, New Mexico, USA (1997) 208–215Google Scholar
  10. 10.
    Schütz, J. and Nübel, R.: Multi-purpose vs. Specific Application: Diagnostic Evaluation of Multilingual Language Technologies. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain (1998) 1251–1254Google Scholar
  11. 11.
    Thurmair, G.: Exchange Interfaces for Translation Tools. In Proceedings of MT Summit VI, San Diego, California, USA (1997) 74–92Google Scholar
  12. 12.
    Thurmair, G., Ritzke, J. and McCormik, S.: The Open Lexicon Interchange Format-OLIF. OTELO Report available on the Web at (1998)
  13. 13.
    TMX Format Specifications. Version 1.0 November-25-1997. Available on the Web at (1997)

Copyright information

© Springer-Verlag Berlin Heidelberg 1998

Authors and Affiliations

  1. 1.IAISaarbrückenGermany

Personalised recommendations