Skip to main content
SpringerLink
Log in

Quality of Spoken Dialog Systems

  • Chapter
  • First Online:
Quality Engineering

  • 245 Accesses

Abstract

After considering systems for the technical support of interpersonal communication in the past two chapters, we will deal with human–machine interaction in this and the following chapter. In order for humans to interact with machines, the latter must be able to recognize and interpret information, as well as output information to humans. Information input and output can be done with the help of different media. The term medium refers to a means of communication (material or device) that uses a particular physical (e.g., acoustic, optical) channel, whereas the term modality refers to the use of that medium for communication, for example, in the form of intonation (spoken language), gaze, gesture, facial expression, etc. Modalities address different senses, for example, in visual, auditory, or haptic perception (includes tactile perception/surface sensitivity, kinesthetic perception/depth sensitivity, temperature perception, as well as pain perception). In this chapter, we will initially restrict ourselves to systems that use the modality “spoken language” for both information input and information output.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliography

  • Bernsen NO, Dybkjær H, Dybkjær L (1998) Designing Interactive Speech Systems: From First Ideas to User Testing. Springer, Berlin

    Book  Google Scholar 

  • Billi R, Castagneri G, Danieli M (1996) Field trial evaluations of two different information inquiry systems. In: Proc. 3rd IEEE Workshop on Interactive Voice Technology for Telecommunications Applications (IVTTA’96), Basking Ridge NJ, S 129–134

    Google Scholar 

  • Bimbot F, Chollet G (1997) Handbook on Standards and Resources for Spoken Language Systems, Mouton de Gruyter, Berlin, Kapitel Assessment of Speaker Verification Systems, S 408–480

    Google Scholar 

  • Boros M, Eckert W, Gallwitz F, Görz G, Hanrieder G, Niemann H (1996) Towards understanding spontaneous speech: Word accuracy vs. concept accuracy. In: Bunnell H, Idsardi W (Hrsg) Proc. 4th Int. Conf. on Spoken Language Processing (ICSLP’96), IEEE, Piscataway NJ, Vol 2, S 1009–1012

    Google Scholar 

  • Carletta J (1996) Assessing agreement on classification tasks: The kappa statistics. Computational Linguistics 22(2):249–254

    Google Scholar 

  • Constantinides PC, Rudnicky AI (1999) Dialog analysis in the Carnegie Mellon Communicator. In: Proc. 6th Europ. Conf. on Speech Communication and Technology (Eurospeech’99), Budapest, Vol 1, S 243–246

    Google Scholar 

  • Cookson S (1988) Final evaluation of VODIS – Voice Operated Database Enquiry System. In: Proc. of SPEECH’88, 7th FASE Symposium, Edinburgh, Vol 4, S 1311–1320

    Google Scholar 

  • Danieli M, Gerbino E (1995) Metrics for evaluating dialogue strategies in a spoken language system. In: Empirical Methods in Discourse Interpretation and Generation. Papers from the 1995 AAAI Symposium (Stanford CA), AAAI Press, Menlo Park CA, S 34–39

    Google Scholar 

  • den Os E, Bloothooft G (1998) Evaluating various spoken dialogue systems with a single questionnaire: Analysis of the ELSNET Olympics. In: Proc. 1st Int. Conf. on Language Resources and Evaluation (LREC’98), Granada, Vol 1, S 51–54

    Google Scholar 

  • Fraser N (1997) Handbook on Standards and Resources for Spoken Language Systems, Mouton de Gruyter, Berlin, Kapitel Assessment of Interactive Systems, S 564–615

    Google Scholar 

  • Gerbino E, Baggia P, Ciaramella A, Rullent C (1993) Test and evaluation of a spoken dialogue system. In: Proc. Int. Conf. Acoustics Speech and Signal Processing (ICASSP’93), IEEE, Piscataway NJ, Vol 2, S 135–138

    Google Scholar 

  • Gibbon D, Moore R, Winski R (Hrsg) (1997) Handbook on Standards and Resources for Spoken Language Systems. Mouton de Gruyter, Berlin

    Google Scholar 

  • Gibbon D, Mertins I, Moore R (Hrsg) (2000) Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation. Kluwer Academic Publ., Boston MA

    Google Scholar 

  • Glass J, Polifroni J, Seneff S, Zue V (2000) Data collection and performance evaluation of spoken dialogue systems: The MIT experience. In: Proc. 6th Int. Conf. on Spoken Language Processing (ICSLP 2000), Beijing, Vol 4, S 1–4

    Google Scholar 

  • Goodine D, Hirschman L, Polifroni J, Seneff S, Zue V (1992) Evaluating interactive spoken language systems. In: Proc. 2nd Int. Conf. on Spoken Language Processing (ICSLP’92), Banff, Vol 1, S 201–204

    Google Scholar 

  • Grice HP (1975) Syntax and Semantics, Vol 3: Speech Acts, Academic Press, New York NY, Kapitel Logic and Conversation, S 41–58

    Google Scholar 

  • Hirschman L, Pao C (1993) The cost of errors in a spoken language system. In: Proc. 3rd Europ. Conf. on Speech Communication and Technology (Eurospeech’93), Berlin, Vol 2, S 1419–1422

    Google Scholar 

  • Hone KS, Graham R (2000) Towards a tool for the subjective assessment of speech system interfaces (SASSI). Natural Language Engineering 6(3-4):287–303

    Article  Google Scholar 

  • ISO Standard 9241 Part 110 (2006) Ergonomics of human-system interaction – Part 110: Dialogue principles. International Organization for Standardization, Geneva

    Google Scholar 

  • ITU-T Rec. P.85 (1994) A Method for Subjective Performance Assessment of the Quality of Speech Voice Output Devices. International Telecommunication Union, Genf

    Google Scholar 

  • ITU-T Rec. P.851 (2003) Subjective Quality Evaluation of Telephone Services Based on Spoken Dialogue Systems. International Telecommunication Union, Genf

    Google Scholar 

  • ITU-T Suppl. 24 to P-Series Rec. (2005) Parameters Describing the Interaction with Spoken Dialogue Systems. International Telecommunication Union, Genf

    Google Scholar 

  • Jack MA, Foster JC, Stentiford FWM (1992) Intelligent dialogues in automated telephone services. In: Proc. 2nd Int. Conf. on Spoken Language Processing (ICSLP’92), Banff, Vol 1, S 715–718

    Google Scholar 

  • Kamm CA, Litman DJ, Walker MA (1998) From novice to expert: The effect of tutorials on user expertise with spoken dialogue systems. In: Proc. 5th Int. Conf. on Spoken Language Processing (ICSLP’98), Sydney, Vol 4, S 1211–1214

    Google Scholar 

  • Lamel L, Minker W, Paroubek P (2000) Towards best practice in the development and evaluation of speech recognition components of a spoken language dialogue system. Natural Language Engineering 6(3–4):305–322

    Article  Google Scholar 

  • Love S, Dutton RT, Foster JC, Jack MA, Stentiford FWM (1994) Identifying salient usability attributes for automated telephone services. In: Proc. 3rd Int. Conf. on Spoken Language Processing (ICSLP’94), Yokohama, Vol 3, S 1307–1310

    Google Scholar 

  • McTear MF (2002) Spoken dialogue technology: Enabling the conversational interface. ACM Computing Surveys 34(1):90–169

    Article  Google Scholar 

  • McTear MF (2004) Spoken Dialogue Technology: Toward the Conversational User Interface. Springer, London

    Book  Google Scholar 

  • Möller S (2005a) Perceptual quality dimensions of spoken dialogue systems: A review and new experimental results. In: Proc. 4th European Congress on Acoustics (Forum Acusticum Budapest 2005), Budapest, S 2681–2686

    Google Scholar 

  • Möller S (2005b) Quality of Telephone-based Spoken Dialogue Systems. Springer, New York NY

    Google Scholar 

  • Möller S (2008) Recent Trends in Discourse and Dialogue, Springer, Dordrecht, Kapitel Evaluating Interactions with Spoken Dialogue Telephone Services, S 69–100

    Google Scholar 

  • Möller S, Smeele P, Boland H, Krebber J (2007) Evaluating spoken dialogue systems according to de-facto standards: A case study. Computer Speech and Language 21:26–53

    Article  Google Scholar 

  • NIST Speech Recognition Scoring Toolkit (2001) Speech Recognition Scoring Toolkit. National Institute of Standards and Technology, http://www.nist.gov/speech/tools, Gaithersburg MD

  • Oulasvirta A, Möller S, Engelbrecht KP, Jameson A (2006) The relationship of user errors to perceived usability of a spoken dialogue system. In: Möller S, Raake A, Jekosch U, Hanisch M (Hrsg) Proc. 2nd ISCA/DEGA Tutorial and Research Workshop on Perceptual Quality of Systems, Int. Speech Comm. Assoc. (ISCA), Berlin, S 61–67

    Google Scholar 

  • Pallett DS, Fourcin A (1997) Survey of the State of the Art in Human Language Technology, Cambridge University Press and Giardini Editori, Pisa, Kapitel Speech Input: Assessment and Evaluation, S 425–429

    Google Scholar 

  • Polifroni J, Hirschman L, Seneff S, Zue V (1992) Experiments in evaluating interactive spoken language systems. In: Proc. DARPA Speech and Natural Language Workshop, Harriman CA, S 28–33

    Google Scholar 

  • Price PJ, Hirschman L, Shriberg E, Wade E (1992) Subject-based evaluation measures for interactive spoken language systems. In: Proc. DARPA Speech and Natural Language Workshop, Harriman CA, S 34–39

    Google Scholar 

  • San-Segundo R, Montero JM, Colás J, Gutiérrez J, Ramos JM, Pardo JM (2001) Methodology for dialogue design in telephone-based spoken dialogue systems: A Spanish train information system. In: Proc. 7th Europ. Conf. on Speech Communication and Technology (Eurospeech 2001 – Scandinavia), Aalborg, Vol 3, S 2165–2168

    Google Scholar 

  • Simpson A, Fraser NM (1993) Black box and glass box evaluation of the SUNDIAL system. In: Proc. 3rd Europ. Conf. on Speech Communication and Technology (Eurospeech’93), Berlin, Vol 2, S 1423–1426

    Google Scholar 

  • Skowronek J (2002) Entwicklung von Modellierungsansätzen zur Vorhersage der Dienstequalität bei der Interaktion mit einem natürlichsprachlichen Dialogsystem. Diplomarbeit (unveröffentlicht), Institut für Kommunikationsakustik, Ruhr-Universität, Bochum

    Google Scholar 

  • Strik H, Cucchiarini C, Kessens JM (2001) Comparing the performance of two CSRs: How to determine the significance level of the differences. In: Proc. 7th Europ. Conf. on Speech Communication and Technology (Eurospeech 2001 – Scandinavia), Aalborg, Vol 3, S 2091–2094

    Google Scholar 

  • van Bezooijen R, van Heuven V (1997) Handbook on Standards and Resources for Spoken Language Systems, Mouton de Gruyter, Berlin, Kapitel Assessment of Synthesis Systems, S 481–563

    Google Scholar 

  • van Leeuwen D, Steeneken H (1997) Handbook on Standards and Resources for Spoken Language Systems, Mouton de Gruyter, Berlin, Kapitel Assessment of Recognition Systems, S 381–407

    Google Scholar 

  • Walker MA, Litman DJ, Kamm CA, Abella A (1997) PARADISE: A framework for evaluating spoken dialogue agents. In: Proc. of the ACL/EACL 35th Ann. Meeting of the Assoc. for Computational Linguistics (Madrid), Morgan Kaufmann, San Francisco CA, S 271–280

    Google Scholar 

  • Walker MA, Litman DJ, Kamm CA, Abella A (1998) Evaluating spoken dialogue agents with PARADISE: Two case studies. Computer Speech and Language 12(3):317–347

    Article  Google Scholar 

  • Zue V, Seneff S, Glass JR, Polifroni J, Pao C, Hazen TJ, Hetherington L (2000) Jupiter: A telephone-based conversational interface for weather information. IEEE Trans Speech and Audio Processing 8(1):85–96

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Quality and Usability Lab, TU Berlin, Berlin, Germany

    Sebastian Möller

Authors
  1. Sebastian Möller
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer-Verlag GmbH, DE, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Möller, S. (2023). Quality of Spoken Dialog Systems. In: Quality Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-65615-0_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-65615-0_7

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-65614-3

  • Online ISBN: 978-3-662-65615-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation