Abstract
Ideally, we might hope to improve the performance of our MT systems by improving the system, but it might be even more important to improve performance by looking for a more appropriate application. A survey of the literature on evaluation of MT systems seems to suggest that the success of the evaluation often depends very strongly on the selection of an appropriate application. If the application is well-chosen, then it often becomes fairly clear how the system should be evaluated. Moreover, the evaluation is likely to make the system look good. Conversely, if the application is not clearly identified (or worse, if the application is poorly chosen), then it is often very difficult to find a satisfying evaluation paradigm. We begin our discussion with a brief review of some evaluation metrics that have been tried in the past and conclude that it is difficult to identify a satisfying evaluation paradigm that will make sense over all possible applications. It is probably wise to identify the application first, and then we will be in a much better position to address evaluation questions. The discussion will then turn to the main point, an essay on how to pick a good niche application for state-of-the-art (crummy) machine translation.
Similar content being viewed by others
References
Brown, P., Cocke, J., Della Pietra, S., Della Pietra, V., Jelinek, F., Lafferty, J., Mercer, R., Rossin, P.: 1990, ‘A Statistical Approach to Machine Translation’,Computational Linguistics 16, 79–85.
Brown, P., Della Pietra, S., Della Pietra, V., Lai J., Mercer, R.: 1991, ‘An Estimate of an Upper Bound for the Entropy of English’, submitted toComputational Linguistics.
Church, K. and Gale, W.: 1991, ‘Concordances for Parallel Text’, Seventh Annual Conference of the UW Centre for the New Oxford English Dictionary and Text Research, Oxford, England.
Henisz-Dostert, B.: 1979, ‘Users' Evaluation of Machine Translation’, in Bozena Henisz-Dostert, R. Ross Macdonald, Michael Zarechnak (eds.),Machine Translation, Mouton Publishers.
Isabelle, P.: 1984, ‘Machine Translation at the TAUM Group’, in King, M. (ed.),Machine Translation Today: The State of the Art, Edinburgh University Press.
Isabelle, P. and Bourbeau, L.: 1985, ‘Taum-Aviation: Its Technical Features and Some Experimental Results’,Computational Linguistics 11, 18–27.
Johnson, R., King, M., and des Tombe, L.: 1985, ‘Eurotra: A Multilingual System under Development’,Computational Linguistics 11, 155–169.
Kay, M.: 1980, ‘The Proper Place of Men and Machines in Language Translation’, unpublished ms., Xerox, Palo Alto, CA.
King, M. and Falkedal, K.: 1990, ‘Using Test Suites in Evaluation of Machine Translation Systems',COLING, pp. 211–216.
Klatt, D.: 1977, ‘Review of the ARPA Speech Understanding Project’Journal of the Acoustical Society of America, reprinted in Waibel, A. and Lee, K. (eds.), 1990,Readings in Speech Recognition, Morgan Kaufmann Publishers, Inc., San Mateo.
Lawson, V.: 1984, ‘Users of Machine Translation System Report Increased Output’,Language Monthly 11, 6–10.
Lehrberger, J. and Bourbeau, L.: 1988,Machine Translation: Linguistic Characteristics of MT Systems and General Methodology of Evaluation, John Benjamins Press.
Lowerre, B. and Reddy, D.: 1980, ‘The Harpy Speech Understanding Systems’, inTrends in Speech Recognition, Prentice Hall, reprinted in Waibel, A. and Lee, K. (eds.), 1990,Readings in Speech Recognition, Morgan Kaufmann Publishers, Inc., San Mateo, California.
Macklovitch, E.: 1986, ‘MT Trial and Errors’, presented at the International Conference on Machine and Machine-Aided Translation, April 7–9, Aston University, Birmingham, United Kingdom.
Macklovitch, E.: 1989, ‘An Off-the-Shelf Workstation for Translators’,Proceedings of the 30th American Translators Conference, Washington DC, 1989.
Macklovitch, E.: 1991, ‘Evaluating Commercial MT Systems’, paper presented at the Evaluators' Form of MT systems, organized by ISSCO at Ste. Croix, Switzerland.
Magnusson-Murray, U.: 1985, ‘Operational Experience of a Machine Translation Service’, in Lawson, V. (ed.),Tools for the Trade, Translating and the Computer, Alden Press, Oxford.
Nagao, M.: 1987, ‘Role of Structural Transformation in a Machine Translation System’, in Nirenburg, S. (ed.),Machine Translation: Theoretical and Methodological Issues, Cambridge University Press, pp. 262–277.
Nagao, M., Tsujii, J-I., and Nakamura, J-I.: 1986, ‘Machine Translation from Japanese to English’,Proceedings of the IEEE 74, 993–1012.
Newell, A., Barnett, J., Forgie, J., Green, C., Klatt, D., Licklider, J., Munson, J., Reddy D., and Woods, W.: 1973,Speech Understanding Systems: Final Report of a Study Groups, North-Holland/American Elsevier, Amsterdam.
Pierce, J., Caroll, J., Hamp, E., Hays, D., Hockett, C., Dettinger, A., Perlis, A.: 1966, ‘Language and Machines: Computers in Translation and Linguistics’, also known as the ALPAC report, National Academy of Sciences Publication 416, Washington D.C.
Schwartz, E.: 1991, ‘A Computer that Recognizes its Master's Voice’,Business Week, June 3, pp. 130–131.
Shannon, C.: 1951, ‘Prediction and Entropy of Printed English’,Bell Systems Technical Journal 30, 50–64.
Tschira, K.: 1985, ‘Looking Back at a Year of German-English MT with Logos’, in Lawon, V. (ed.),Tool for the Trade, Translating and the Computer, Alden Press, Oxford.
Tucker, A.B.: 1987, ‘Current Strategies in Machine Translation Research and Development’, in Nirenburg, S. (ed.),Machine Translation: Theoretical and Methodological Issues, Cambridge University Press, pp. 22–41.
Van Sylpe, G.: 1979, ‘Evaluation of the f1978 Version of the SYSTRAN English-French Automatic System of the Commission of the European Communities’,The Incorporated Linguistics 18, 86–89.
Wagner, E.: 1985, ‘Rapid Post-Editing of Systran’, in Lawson, V. (ed.),Tools for the Trade, Translating and the Computer, Alden Press, Oxford.
Zarechnak: 1979, ‘This History of Machine Translation’, in Bozena Henisz-Dostert, R. Ross Macdonald, Michael Zarechnak (eds.),Machine Translation, Mouton Publishers.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Church, K.W., Hovy, E.H. Good applications for crummy machine translation. Mach Translat 8, 239–258 (1993). https://doi.org/10.1007/BF00981759
Issue Date:
DOI: https://doi.org/10.1007/BF00981759