Real Users and Real Dialog Systems: The Hard Challenge for SDS

Conference paper


Much of the research done in our community is based on developing spoken dialog systems and testing various techniques within those dialog systems. Because it makes it easier to deal with our experimental conditions, many of our tests and studies involve controlled (paid or volunteered) users. However, we have seen in a number of studies that these controlled users do not use the system in the same way as those for whom the system was actually designed. Sometimes the difference between the real user, who wants the information the spoken dialog system is providing or who wants to give information to it, and the controlled user, who is acting under some direction, is not that different. Certainly in some circumstances it is necessary to use the latter. But, since state-of-the-art systems have become increasingly reliant on large amounts of user data to train their models of behavior, it is critical that the user behavior we train on is real user behavior. This paper describes the issues that arise when building a spoken dialog system for real users. The goal is to provide both a service to the user and a realistic spoken dialog system (SDS) research platform.


Live System Surrogate Data Control User Real User Port Authority 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Ai, H., Raux, A., Bohus, D., Eskenazi, M., Litman, D.: Comparing spoken dialog corpora collected with recruited subjects versus real users. In: SIGDial, Columbus, OH (2008)Google Scholar
  2. 2.
    Black, A., Burger, S., Conkie, A., Hastie, H., Keizer, S., Lemon, O., Merigaud, N., Parent, G., Schubiner, G., Thomson, B., Williams, J., Yu, K., Young, S., Eskenazi, M.: Spoken dialog challenge 2010: Comparison of live and control test results. In: SIGDial, Portland, OR (2012)Google Scholar
  3. 3.
    Bohus, D., Langner, B., Raux, A., Black, A., Eskenazi, M., Rudnicky, A.: Online supervised learning of non-understanding recovery policies. In: SLT-2006, Palm Beach, Aruba (2006)Google Scholar
  4. 4.
    Fandrianto, A., Eskenazi, M.: Prosodic entrainment in an information-driven dialog system. In: Interspeech, Portland, OR (2012)Google Scholar
  5. 5.
    Lee, S., Eskenazi, M.: An unsupervised approach to user simulation: Toward self-improving dialog systems. In: Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 50–59. Association for Computational Linguistics, Seoul, South Korea (2012). URL
  6. 6.
    McGraw, I., Gruenstein, A., Sutherland, A.: A self-labeling speech corpus: Collecting spoken words with an online educational game. In: Interspeech, Brighton, UK (2009)Google Scholar
  7. 7.
    Raux, A., Bohus, D., Langner, B., Black, A., Eskenazi, M.: Doing research on a deployed spoken dialogue system: one year of let’s go experience. In: Interspeech, Pittsburgh, PA (2006)Google Scholar
  8. 8.
    Rojas-Barahona, L., Lorenzo, A., Gardent, C.: Building and exploiting a corpus of dialog interactions between french speaking virtual and human agents. In: LREC, Istanbul, Turkey (2012)Google Scholar
  9. 9.
    Stoyanchev, S., Stent, A.: Predicting concept types in user corrections in dialog. In: EACL Workshop on Semantic Representation of Spoken Language, Athens, Greece (2009)Google Scholar
  10. 10.
    Swartout, W., Traum, D.R., Artstein, R., oren, D.N., Debevec, P., Bronnenkant, K., Williams, J., Shrikanth Narayanan, A.L., Piepol, D., Lane, C., Morie, J., iti Aggarwal, P., Liewer, M., Chiang, J.Y., Gerten, J., hu, S.C., White, K.: Ada and grace: Toward realistic and engaging virtual museum guides. In: IVA 2010, Philadelphia (2010)Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Language Technologies InstituteCarnegie Mellon UniversityPAUSA

Personalised recommendations