Abstract
This paper describes a task-based evaluation methodology appropriate for dialogue systems such as the TRAINS-95 system, where a human and a computer interact and collaborate to solve a given problem. In task-based evaluations, techniques are measured in terms of their effect on task performance measures such as how long it takes to develop a solution using the system, and the quality of the final plan produced. We report recent experiment results which explore the effect of word recognition accuracy on task performance.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Funding was gratefully received from NSF under Grant IRI-90-13160 and from ONR/DARPA under Grant N00014-92-J-1512. Many thanks to George Ferguson for developing the on-line tutorial, Eric Ringger for compiling the word recognition accuracy figures, Amon Seagull for advice on statistical measures, and Peter Heeman for numerous helpful comments. Thanks also to Mike Tanenhaus and Joy Hanna for their suggestions on the experimental design.
This is a preview of subscription content, log in via an institution.
Preview
Unable to display preview. Download preview PDF.
References
J. F. Allen, G. Ferguson, B. Miller, and E. Ringger. Spoken Dialogue and Interactive Planning. In Proceedings of the ARPA SLST Workshop, San Mateo California, January 1995. Morgan Kaufmann.
M. Boros, W. Eckert, F. Gallwitz, G. Görz, G. Hanrieder, H. Niemann. Towards Understanding Spontaneous Speech: Word Accuracy Vs. Concept Accuracy. In Proceedings of the International Conference on Spoken Language Processing, Philadelphia, Pennsylvania, October 1996.
P. Cohen and S. Oviatt. The Role of Voice Input for Human-Machine Communication. In Proceedings of the National Academy of Sciences, 1994.
L. Hirschman, M. Bates, D. Dahl, W. Fisher, J. Garofolo, D. Pallet, K. Hunicke-Smith, P. Price, A. Rudnicky and E. Tzoukermann. Multi-Site Data Collection and Evaluation in Spoken Language Understanding. In Proceedings of the ARPA Human Language Technology Workshop, Princeton, New Jersey, March 1993. Morgan Kaufmann.
X. D. Huang, F. Alleva, H.W. Hon, M. Y. Hwang, K. F. Lee, and R. Rosenfeld. The Sphinx-II Speech Recognition System: An Overview. Computer, Speech and Language, 1993.
S. Oviatt and P. Cohen. The Contributing Influence of Speech and Interaction on Human Discourse Patterns. In J. W. Sullivan and S. W. Tyler (eds), Intelligent User Interfaces. New York, New York. 1991. Addison-Wesley.
J. Polifroni, L. Hirschman, S. Seneff, and V. Zue. Experiments in Evaluating Interactive Spoken Language Systems. In Proceedings of the DARPA Speech and Natural Language Workshop, Harriman, New York, February 1992. Morgan Kaufmann.
E. Ringger and J. F. Allen. Error Correction Via A Post-Processor For Continuous Speech Recognition. Proceedings of ICASSP-96, Atlanta Georgia, May 1996.
A. Rudnicky. Mode Preferences in a Simple Data Retrieval Task. In Proceedings of the ARPA Human Language Technology Workshop, Princeton, New Jersey, March 1993. Morgan Kaufmann.
E. Shriberg, E. Wade, and P. Price. Human-Machine Problem Solving Using Spoken Language Systems (SLS): Factors Affecting Performance and User Satisfaction. In Proceedings of the DARPA Speech and Natural Language Workshop, Harriman, New York, February 1992. Morgan Kaufmann.
R. Smith and R. D. Hipp. Spoken Natural Language Dialog Systems: A Practical Approach, Oxford University Press. 1994.
S. Walter. Neal-Montgomery NLP System Evaluation Methodology. In Proceedings of the DARPA Speech and Natural Language Workshop, Harriman, New York, February 1992. Morgan Kaufmann.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1997 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sikorski, T., Allen, J.F. (1997). A task-based evaluation of the TRAINS-95 dialogue system. In: Maier, E., Mast, M., LuperFoy, S. (eds) Dialogue Processing in Spoken Language Systems. DPSLS 1996. Lecture Notes in Computer Science, vol 1236. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-63175-5_48
Download citation
DOI: https://doi.org/10.1007/3-540-63175-5_48
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-63175-0
Online ISBN: 978-3-540-69206-5
eBook Packages: Springer Book Archive