Abstract
In this paper we report on ongoing verification and validation work within the MULTIDOC project. This project is situated in the field of multilingual automotive product documentation. One central task is the evaluation of existing off-the-shelf and research based language technology (LT) products and components for the purpose of supporting or even reorganising the documentation production chain along three diagnostic dimensions: the process proper, the documentation quality and the translatability of the process output. In this application scenario, LT components shall control and ensure that predefined quality criteria are applicable and measurable to the documentation end-product as well as to the information objects that form the basic building blocks of the end-product. In this scenario, multilinguality is of crucial importance. It shall be introduced or prepared, and maintained as early as possible in the documentation workflow to ensure a better and faster translation process. A prerequisite for the evaluation process is the thorough definition of these dimensions in terms of user quality requirements and LT developer quality requirements. In our approach, we define the output quality of the whole documentation process as the pivot where user requirements and developer requirements shall meet. For this, it turned out that a so-called “braided” diagnostic evaluation is very well suited to cover both views. Since no generally approved standards or even valid specifications for standards exist for the evaluation of LT products, we have adjusted existing standards for the evaluation of software products, in particular ISO 9001, ISO 9000-3, ISO/IEC 12119, ISO 9004 and ISO 9126. This is feasible because an LT product consists of a software part and a lingware part. The adaptation had to be accomplished for the latter part.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
EAGLES: Evaluation of Natural Language Processing System. Final Report, EAGLES Document EAG-EWG-PR.2, Geneva, Switzerland (1995)
Haller, J.: MULTILINT-Multilingual Documentation with Linguistic Intelligence. In: Proceedings of ‘Translating and the Computer’, ASLIB, London, Great Britain (1996)
Haller, J. and Schütz, J.: Integration linguistischer Intelligenz in die multilinguale technische Dokumentation. In Proceedings of EUROMAP Forum’ sprache ohne Grenzen’, M:unchen, Germany (1997)
Maas, H.D.: Multilinguale Textverarbeitung mit MPRO. In: Lobin, G., Lohse, H. Piotrowski, S and Poláková, E. (Eds.): Europäische Kommunikationskybernetik heute und morgen, KoPäd, München, Germany (1998) 167–173
Nübel, R.: End-to-End Evaluation in Verbmobil I. In: Proceedings of Machine Translation Summit VI, San Diego, California, USA (1997) 232–239
OpenTag-Formal Specifications. Version 1.1b April-22-1998, Last edit: May-01-1998. Available on the Web at http://www.opentag.org/otspecs.htm (1998)
Schütz, J.: Language Engineering-Fixing Positions. IAI Memo 0695, Saarbrücken, Germany. Available on the Web at http://www.iai.uni-sb.de/memos.html (1995)
Schütz, J.: Combining Language Technology and Web Technology to Streamline an Automotive Hotline Support Service. In: Proceedings of AMTA 96, Montreal, Canada (1996) 180–189
Schütz, J.: Utilizing Evaluation in Networked Machine Translation. In: Proceedings of the 7th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI) 1997, Santa Fe, New Mexico, USA (1997) 208–215
Schütz, J. and Nübel, R.: Multi-purpose vs. Specific Application: Diagnostic Evaluation of Multilingual Language Technologies. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain (1998) 1251–1254
Thurmair, G.: Exchange Interfaces for Translation Tools. In Proceedings of MT Summit VI, San Diego, California, USA (1997) 74–92
Thurmair, G., Ritzke, J. and McCormik, S.: The Open Lexicon Interchange Format-OLIF. OTELO Report available on the Web at http://www.otelo.lu (1998)
TMX Format Specifications. Version 1.0 November-25-1997. Available on the Web at http://www.lisa.org/tmx/tmx.htm (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1998 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Schütz, J., Nübel, R. (1998). Evaluating Language Technologies: The MULTIDOC Approach to Taming the Knowledge Soup. In: Farwell, D., Gerber, L., Hovy, E. (eds) Machine Translation and the Information Soup. AMTA 1998. Lecture Notes in Computer Science(), vol 1529. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-49478-2_22
Download citation
DOI: https://doi.org/10.1007/3-540-49478-2_22
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-65259-5
Online ISBN: 978-3-540-49478-2
eBook Packages: Springer Book Archive