Advertisement

The Fourth Dialog State Tracking Challenge

  • Seokhwan Kim
  • Luis Fernando D’Haro
  • Rafael E. Banchs
  • Jason D. Williams
  • Matthew Henderson
Chapter
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 427)

Abstract

Dialog state tracking is one of the key sub-tasks of dialog management , which defines the representation of dialog states and updates them at each moment on a given on-going conversation. To provide a common test bed for this task, three dialog state tracking challenges have been completed. In this fourth challenge, we focused on dialog state tracking on human-human dialogs. The challenge received a total of 24 entries from 7 research groups. Most of the submitted entries outperformed the baseline tracker based on string matching with ontology contents. Moreover, further significant improvements in tracking performances were achieved by combining the results from multiple trackers. In addition to the main task, we also conducted pilot track evaluations for other core components in developing modular dialog systems using the same dataset.

Keywords

Dialog state tracking Dialog management Challenge overview 

References

  1. 1.
    Williams, J., Raux, A., Ramachandran, D., Black, A.: The dialog state tracking challenge. In: Proceedings of the SIGDIAL 2013 Conference, pp. 404–413 (2013)Google Scholar
  2. 2.
    Henderson, M., Thomson, B., Williams, J.: The second dialog state tracking challenge. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, p. 263 (2014)Google Scholar
  3. 3.
    Henderson, M., Thomson, B., Williams, J.D.: The third dialog state tracking challenge. In: Proceedings of the 2014 IEEE Spoken Language Technology Workshop (SLT), pp. 324–329. IEEE (2014)Google Scholar
  4. 4.
    Kim, S., D’Haro, L.F., Banchs, R.E., Williams, J., Henderson, M.: Dialog state tracking challenge 4 handbook. http://www.colips.org/workshop/dstc4/Handbook_DSTC4.pdf (2015)
  5. 5.
    Smith, R.W.: Comparative error analysis of dialog state tracking. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, p. 300 (2014)Google Scholar
  6. 6.
    Lee, S., Eskenazi, M.: Recipe for building robust spoken dialog state trackers: dialog state tracking challenge system description. In: Proceedings of the SIGDIAL 2013 Conference, pp. 414–422 (2013)Google Scholar
  7. 7.
    Yeh, A.: More accurate tests for the statistical significance of result differences. In: Proceedings of the 18th Conference on Computational linguistics, vol. 2. pp. 947–953. Association for Computational Linguistics (2000)Google Scholar
  8. 8.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational linguistics, pp. 311–318. Association for Computational Linguistics (2002)Google Scholar
  9. 9.
    Banchs, R.E., D’Haro, L.F., Li, H.: Adequacy-fluency metrics: evaluating mt in the continuous space model framework. IEEE/ACM Trans. Audio Speech Lang. Process. 23(3), 472–482 (2015)CrossRefGoogle Scholar
  10. 10.
    Kim, S., D’Haro, L.F., Banchs, R.E., Williams, J., Henderson, M.: Dialog state tracking challenge 4 pilot task guidelines. http://www.colips.org/workshop/dstc4/DSTC4_pilot_tasks.pdf (2015)

Copyright information

© Springer Science+Business Media Singapore 2017

Authors and Affiliations

  • Seokhwan Kim
    • 1
  • Luis Fernando D’Haro
    • 1
  • Rafael E. Banchs
    • 1
  • Jason D. Williams
    • 2
  • Matthew Henderson
    • 3
  1. 1.Institute for Infocomm ResearchFusionopolisSingapore
  2. 2.Microsoft ResearchRedmondUSA
  3. 3.GoogleMountain ViewUSA

Personalised recommendations