Spoken Dialogue Systems Evaluation

This chapter first provides a brief introduction to evaluation methods and criteria and then presents two very different spoken dialogue research prototype systems and their evaluation. The first prototype is the non-task-oriented, multimodal Hans Christian Andersen (HCA) system for edutainment, the second prototype is the task-oriented, multimodal SENECA onboard system in the car. The systems were tested with representative users in the laboratory and in the field, respectively. For both systems we describe rationale for the chosen evaluation method, evaluation process, evaluation criteria, and evaluation results.