Abstract
This paper presents a system built for improving the evaluation process of multimodal dialogue systems by providing a graphical representation of event-based recording data. Low-level information from recordings can visually be combined to form higher-level actions using pattern definitions to create a hierarchy of actions. Actions are visually represented by a multitrack player-like view where all modalities of the recording can be watched and manipulated. But besides tagging and derivation of higher-level actions, further help is provided by a correlation analysis and an export of charts and tables to Microsoft Excel. The tool is tested on recording data and questionnaire answers obtained from Wizard of Oz (WOZ) experiments performed within the DICIT project.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Beringer, N., Hans, S., Louka, K., Tang, J.: How to relate User Satisfaction and System Performance in Multimodal Dialogue Situations - a Graphical Approach. In: International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems, pp. 8–14 (2002a)
Beringer, N., Kartal, U., Louka, K., Schiel, F., Türk, U.: PROMISE - A Procedure for Multimodal Interactive System Evaluation. In: 3rd International Conference on Language Resources and Evaluation (LREC), pp. 77–80 (2002b)
DICIT (2007), http://dicit.fbk.eu/
Dybkjær, L., Bernsen, N.: Usability Evaluation in Spoken Language Dialogue Systems. In: Workshop on Evaluation for Language and Dialogue Systems, pp. 1–10 (2001)
Fleischmann, T.: Model Based HMI Specification in an Automotive Context. In: Smith, M.J., Salvendy, G. (eds.) HCII 2007. LNCS, vol. 4557, pp. 31–39. Springer, Heidelberg (2007)
Goronzy, S., Mochales, R., Beringer, N.: Developing speech dialogs for multimodal HMIs using finite state machines. In: 9th International Conference on Spoken Language Processing (Interspeech), CD-ROM (2006)
Kipp, M.: ANVIL - A Generic Annotation Tool for Multimodal Dialogue. In: 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367–1370 (2001)
Milde, J.-T., Gut, U.: The TASX-environment: an XML-based Toolset for Time Aligned Speech Corpora. In: 3rd International Conference on Language Resources and Evaluation (LREC), pp. 1922–1927 (2002)
Möller, S.: Parameters for Quantifying the Interaction with Spoken Dialogue Telephone Services. In: 6th SIGdial Workshop on Discourse and Dialogue, pp. 166–177 (2005)
Walker, M., Kamm, C., Litman, D.: Towards Developing General Models of Usability with PARADISE. Nat. Lang. Eng. 6(3-4), 363–377 (2000)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wesseling, H., Bezold, M., Beringer, N. (2008). Automatic Evaluation Tool for Multimodal Dialogue Systems. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds) Perception in Multimodal Dialogue Systems. PIT 2008. Lecture Notes in Computer Science(), vol 5078. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-69369-7_36
Download citation
DOI: https://doi.org/10.1007/978-3-540-69369-7_36
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-69368-0
Online ISBN: 978-3-540-69369-7
eBook Packages: Computer ScienceComputer Science (R0)