User Modeling and User-Adapted Interaction

, Volume 12, Issue 2–3, pp 111–137

Designing and Evaluating an Adaptive Spoken Dialogue System

  • Diane J. Litman
  • Shimei Pan
Article

Abstract

Spoken dialogue system performance can vary widely for different users, as well for the same user during different dialogues. This paper presents the design and evaluation of an adaptive version of TOOT, a spoken dialogue system for retrieving online train schedules. Based on rules learned from a set of training dialogues, adaptive TOOT constructs a user model representing whether the user is having speech recognition problems as a particular dialogue progresses. Adaptive TOOT then automatically adapts its dialogue strategies based on this dynamically changing user model. An empirical evaluation of the system demonstrates the utility of the approach.

adaptive spoken dialogue systems hypothesis testing for the effectiveness of adaptations PARADISE for evaluating performance measures speech recognition user model acquisition via machine learning 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bell, L. and Gustafson, J.: 2000, Positive and Negative User Feedback in a Spoken Dialogue Corpus. In: Proc. 6th International Conference of Spoken Language Processing (ICSLP). Beijing, China, pp. 589–592.Google Scholar
  2. Bouwman, A. G., Sturm, J. and Boves, L.: 1999, Incorporating confidence measures in the Dutch train timetable information system developed in the ARISE project. In: Proc. International Conference on Acoustics, Speech and Signal Processing, Vol. 1. Phoenix, pp. 493–496.Google Scholar
  3. Chin, D. N.: 2001, Empirical Evaluation of User Models and User-Adapted Systems. User Modeling and User-Adapted Interaction, pp. 181–194.Google Scholar
  4. Chu-Carroll, J.: 2000, MIMIC: An Adaptive Mixed Initiative Spoken Dialogue System for Information Queries. In: Proc. Applied Natural Language Processing (ANLP), pp. 97–104.Google Scholar
  5. Chu-Carroll, J. and Nickerson, J. S.: 2000, Evaluating Automatic Dialogue Strategy Adaptation for a Stoken Dialogue System. In: Proc. 1st Conference of the North American Chapter of the Association for Coputational Linguistics (NAACL), pp. 202–209.Google Scholar
  6. Cohen, P.: 1995, Empirical Methods for Artificial Intelligence. MIT Press, Boston.Google Scholar
  7. Cohen, W.: 1996, Learning trees and rules with set-valued features. In: Proc. 13th National Conference on Artificial Intelligence (AAAI), pp. 709–716.Google Scholar
  8. Danieli, M. and Gerbino, E.: 1996, Metrics for Evaluating Dialogue Strategies in a Spoken Language System. In: Proc. AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pp. 34–39.Google Scholar
  9. Hirasawa, J., Miyazaki, N., Nakano, M. and Aikawa, K.: 2000, New Feature Parameters For Detecting Misunderstandings in a Spoken Dialogue System. In: Proc. 6th International Conference of Spoken Language Processing (ICSLP), Vol. 2, Beijing, China, pp. 154–157.Google Scholar
  10. Kamm, C., Litman, D. and Walker, M.: 1998, From Novice to Expert: The Effect of Tutorials on User Expertise with Spoken Dialogue Systems. In: Proc ICSLP, pp. 1211–1214.Google Scholar
  11. Kamm, C., Narayanan, S., Dutton, D. and Ritenour, R.: 1997, Evaluating Spoken Dialog Systems for Telecommunication Services. In: Proc. European Conference on Speech Communication and Technology (EUROSPEECH), pp. 22–25.Google Scholar
  12. Krahmer, E., Swerts, M., Theune, M. and Weegels, M.: 1999, Error Spotting in Human–Machine Interactions. In: Proc. European Conference on Speech Communication and Technology (EUROSPEECH), pp. 1423–1426.Google Scholar
  13. Levin, E. and Pieraccini, R.: 1997, A Stochastic Model of Computer–Human Interaction for Learning Dialogue Strategies. In: Proc. European Conference on Speech Communication and Technology (EUROSPEECH), pp. 1883–1886.Google Scholar
  14. Levow, G.-A: 1998, Characterizing and Recognizing Spoken Corrections in Human–Computer Dialogue. In: Proc. 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING-ACL), pp. 736–742.Google Scholar
  15. Litman, D., Hirschberg, J. and Swerts, M.: 2000a, Predicting Automatic Speech Recognition Performance Using Prosodic Cues. In: Proc. 1st Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 218–225.Google Scholar
  16. Litman, D., Pan, S. and Walker, M.: 1998, Evaluating Response Strategies in a Web-Based Spoken Dialogue Agent. In: Proc. ACL/COLING, pp. 780–786.Google Scholar
  17. Litman, D. J.: 1998, Predicting Speech Recognition Performance from Dialogue Phenomena. Presented at the American Association for Artificial Intelligence Spring Symposium Series on Applying Machine Learning to Discourse Processing.Google Scholar
  18. Litman, D. J., Kearns, M. S., Singh, S. and Walker, M. A.: 2000b, Automatic Optimization of Dialogue Management. In: Proc. of COLING 2000.Google Scholar
  19. Litman, D. J. and Pan, S.: 1999, Empirically Evaluating an Adaptable Spoken Dialogue System. In: Proc. 7th International Conference on User Modeling (UM), pp. 55–64.Google Scholar
  20. Litman, D. J., Walker, M. A. and Kearns, M. J.: 1999: Automatic Detection of Poor Speech Recognition at the Dialogue Level. In: Proc. 37th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 309–316.Google Scholar
  21. Monge, P. and Cappella, J. (eds): 1980, Multivariate Techniques in Human Communication Research. Academic Press, New York.Google Scholar
  22. Polifroni, J., Hirschman, L., Seneff, S. and Zue, V.: 1992, Experiments in Evaluating Interactive Spoken Language Systems. In: Proc. DARPA Speech and NL Workshop, pp. 28–33.Google Scholar
  23. Shriberg, E., Wade, E. and Price, P.: 1992, Human–Machine Problem Solving Using Spoken Language Systems (SLS): Factors Affecting Performance and User Satisfaction. In: Proc. DARPA Speech and NL workshop, pp. 419–424.Google Scholar
  24. Smith, R.W.: 1998, An Evaluation of Strategies for Selectively Verifying Utterance Meanings in Spoken Natural Lanauage Dialog. International Journal of Human–Computer Studies 48, 627–647.Google Scholar
  25. Strachan, L., Anderson, J., Sneesby, M. and Evans, M.: 1997, Pragmatic UserModelling in a Commercial Software System. In: Proc. UM97, pp. 189–200.Google Scholar
  26. van Zanten, G. V.: 1999, User Modelling in Adaptive Dialogue Management. In: Proc. European Conference on Speech Communication and Technology (EUROSPEECH), pp. 1183–1186.Google Scholar
  27. Walker, M., Fromer, J. and Narayanan, S.: 1998a, Learning Optimal Dialogue Strategies. A Case Study of a Spoken Dialogue Agent for E-mail. In: Proc. 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics (COLING-ACL), pp. 1345–1352.Google Scholar
  28. Walker, M., Hindle, D., Fromer, J., Fabbrizio,G. D. and Mestel, C.: 1997a, Evaluating Competing Agent Strategies for a Voice E-mail Agent. In: Proc. European Conference on Speech Communication and Technology (EUROSPEECH), pp. 22–25.Google Scholar
  29. Walker, M., Langkilde, I., Wright, J., Gorin, A. and Litman, D.: 2000a. Learning to Predict Problematic Situations in a Spoken Dialogue System: Experiments with HowMay I Help You?. In: Proceedings of the North American Meeting of the Association for Computational Linguistics, pp. 210–217.Google Scholar
  30. Walker, M., Litman, D., Kamm, C. and Abella, A.: 1997b, PARADISE: A General Framework for Evaluating Spoken Dialogue Agents. In: Proc. ACL/EACL, pp. 271–280.Google Scholar
  31. Walker, M., Litman, D., Kamm, C. and Abella, A.:1998b, Evaluating Spoken Dialogue Agents with PARADISE Two Case Studies. Computer Speech and Language, 12(3), pp. 317–347.Google Scholar
  32. Walker, M. A., Kamm, C. A. and Litman, D. J. 2000b, Towards Developing General Models of Usability with PARADISE. Natural Language Engineering: Special Issue on Best Practice in Spoken Dialogue Systems.Google Scholar
  33. Weiss, S. M. and Kulikowski, C.: 1991, Computer Systems That Learn: Classification and Prediction Methods from Statistics, Neural Nets, Machine Learning, and Expert Systems. San Mateo, CA: Morgan Kaufmann.Google Scholar
  34. Zeljkovic, I.: 1996, Decoding Optimal State Sequences with Smooth State Likelihoods. In: Proc. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 129–132.Google Scholar

Copyright information

© Kluwer Academic Publishers 2002

Authors and Affiliations

  • Diane J. Litman
    • 1
  • Shimei Pan
    • 2
  1. 1.Computer Science Department and LRDCUniversity of PittsburghPittsburghUSA
  2. 2.IBM T.J. Watson Research CenterHawthorneUSA

Personalised recommendations