Skip to main content
SpringerLink
Log in
Book cover

Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop pp 343–355Cite as

Design and Implementation of a Toolkit for Evaluation of Spoken Dialogue Systems Designed for AmI Environments

  • Conference paper
  • First Online:

  • 456 Accesses

Abstract

This paper presents a new toolkit for the evaluation of spoken dialogue systems in Ambient Intelligence domains. It describes the design and implementation of the toolkit, which consists in several tools that can be employed to process corpora of recorded spoken interaction and automatically obtain statistical measures in order to assess and improve these systems. The toolkit has been applied to evaluate the Mayordomo multimodal dialogue system developed in our laboratory, which enables automatic control of home appliances. This evaluation was useful to detect problems in the dialogue system suggesting which modules must be improved.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ábalos, N., Espejo, G., López-Cózar, R., Callejas, Z., Griol, D.: AMultimodal Dialogue System for an Ambient Intelligent Application in Home Environments. LNAI 6231, 491–498 (2010)

    Google Scholar 

  2. Araki, M., Doshita, S.: Automatic evaluation environment for spoken dialogue systems. In: Proc. of ECAI Workshop on Dialogue Processing in Spoken, pp. 183–194 (1997).

    Google Scholar 

  3. Boros, M., Eckert, W., Gallwitz, F., Gorz, G:. Towards understanding spontaneous speech: Word accuracy vs. concept accuracy. In: Proc. of International Conference on Spoken Language (ICSLP’96), pp. 1009–1012 (1996)

    Google Scholar 

  4. Espejo, G., Ábalos, N., López-Cózar, R., Callejas, Z., Griol, D.: Mayordomo: Ambient Intelligence in a Home Environment. In: Proc. of HCIAmI’10, (2010)

    Google Scholar 

  5. IST Advisory Group. The European Union Report, Scenarios for Ambient Intelligence in 2010, (2001)

    Google Scholar 

  6. López-Cózar, R., Araki, M.: Spoken, Multilingual and Multimodal Dialogue Systems. John Wiley and Sons Publishers (2005)

    Google Scholar 

  7. López-Cózar, R., de la Torre, A., Segura, J. C., Rubio, A. J.: Assessment of Dialogue Systems by Means of a New Simulation Technique. Speech Communication 40(3), 387–407 (2003)

    Article  Google Scholar 

  8. López-Cózar, R., Griol, D., Espejo, G., Callejas, Z., Ábalos, N.: Towards Fine-Grain User-Simulation For Spoken Dialogue Systems. Spoken Dialogue Systems Technology and Design. pp. 53–81. Springer Science and Business Media (2011)

    Google Scholar 

  9. McTear, M.F.: Spoken Dialogue Technology: Towards the Conversational User Interface. Springer (2004)

    Google Scholar 

  10. Moller, S.: Assessment and Evaluation of Speech-Based Interactive Systems: From Manual Annotation to Automatic Usability Evaluation. Speech Technology. Springer (2010)

    Google Scholar 

  11. Moller, S., Englert, R., Engelbrecht, K., Hafner, V., Jameson, A., Oulasvirta, A., Raake, A., Reithinger, N. MeMo: Towards Automatic Usability Evaluation of Spoken Dialogue Services by User Error Simulations. In: Proc. of Interspeech, pp. 1786–1789 (2006)

    Google Scholar 

  12. Danieli, M. and Gerbino, E.: Metrics for evaluating dialogue strategies in a spoken language system. In: Proc. of the AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pp. 34–39 (1995)

    Google Scholar 

  13. Walker, M.: Experimentally evaluating communicative strategies: The effect of the task. In: Proc. of the Conference of the American Association of Artificial Intelligence (AAAI’94), pp. 86–93 (1994)

    Google Scholar 

Download references

Acknowledgements

Research funded by the Spanish Ministry of Science and Innovation, under the project ASIES TIN2010-17344: Adaptation in Smart Environments and Social Networks to Assist People with Special Needs.

Author information

Authors and Affiliations

  1. Dept. of LSI, CITIC-UGR, University of Granada, Granada, Spain

    Nieves Ábalos, Gonzalo Espejo, Ramón López-Cózar & Zoraida Callejas

  2. Dept. of Computer Science, Carlos III University of Madrid, Madrid, Spain

    David Griol

Authors
  1. Nieves Ábalos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gonzalo Espejo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ramón López-Cózar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zoraida Callejas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Griol
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nieves Ábalos .

Editor information

Editors and Affiliations

  1. , Dept. of Languages and Computer Systems, University of Granada, Granada, 18071, Spain

    Ramón López-Cózar Delgado

  2. , Dept. of Computer Science & Engineering, Waseda University, Okubo 3-4-1, Tokyo, 169-8555, Japan

    Tetsunori Kobayashi

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer Science+Business Media, LLC

About this paper

Cite this paper

Ábalos, N., Espejo, G., López-Cózar, R., Callejas, Z., Griol, D. (2011). Design and Implementation of a Toolkit for Evaluation of Spoken Dialogue Systems Designed for AmI Environments. In: Delgado, RC., Kobayashi, T. (eds) Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-1335-6_33

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-1335-6_33

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-1334-9

  • Online ISBN: 978-1-4614-1335-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation