User-aware dialogue management policies over attributed bi-automata

  • Manex Serras
  • María Inés Torres
  • Arantza del Pozo
Original Article


Designing dialogue policies that take user behavior into account is complicated due to user variability and behavioral uncertainty. Attributed probabilistic finite-state bi-automata (A-PFSBA) have proven to be a promising framework to develop dialogue managers that capture the users’ actions in its structure and adapt to them online, yet developing policies robust to high user uncertainty is still challenging. In this paper, the theoretical A-PFSBA dialogue management framework is augmented by formally defining the notation of exploitation policies over its structure. Under such definition, multiple path-based policies are implemented, those that take into account external information and those which do not. These policies are evaluated on the Let’s Go corpus, before and after an online learning process whose goal is to update the initial model through the interaction with end users. In these experiments the impact of user uncertainty and the model structural learning is thoroughly analyzed.


Dialogue systems User adaptation Attributed bi-automata Dialogue management Path-based policies 



This work has been partially founded by the Spanish Minister of Science under Grants TIN2014-54288-C4-4-R and TIN2017-85854-C4-3-R and by the European Commission H2020 SC1-PM15 EMPATHIC project, RIA Grant 69872.


  1. 1.
    Bohus D, Rudnicky AI (2009) The ravenclaw dialog management framework: architecture and systems. Comput Speech Lang 23(3):332–361CrossRefGoogle Scholar
  2. 2.
    Cuayáhuitl H (2017) Simpleds: a simple deep reinforcement learning dialogue system. In: Dialogues with social robots, Springer, Berlin, pp 109–118Google Scholar
  3. 3.
    Gašić M, Jurčíček F, Keizer S, Mairesse F, Thomson B, Yu K, Young S (2010) Gaussian processes for fast policy optimisation of pomdp-based dialogue managers. In: Proceedings of the 11th annual meeting of the special interest group on discourse and dialogue, association for computational linguistics, pp 201–204Google Scholar
  4. 4.
    Ghigi F, Torres MI (2015) Decision making strategies for finite-state bi-automaton in dialog management. In: Natural language dialog systems and intelligent assistants, Springer, Berlin, pp 209–221Google Scholar
  5. 5.
    Gorin AL, Riccardi G, Wright JH (1997) How may i help you? Speech Commun 23(1–2):113–127CrossRefzbMATHGoogle Scholar
  6. 6.
    Griol D, Callejas Z, López-Cózar R (2010) Statistical dialog management methodologies for real applications. In: Proceedings of the 11th annual meeting of the special interest group on discourse and dialogue, association for computational linguistics, pp 269–272Google Scholar
  7. 7.
    Hurtado LF, Griol D, Sanchis E, Segarra E (2007) A statistical user simulation technique for the improvement of a spoken dialog system. In: Iberoamerican congress on pattern recognition, Springer, Berlin, pp 743–752Google Scholar
  8. 8.
    Hurtado LF, Planells J, Segarra E, Sanchis E, Griol D (2010) A stochastic finite-state transducer approach to spoken dialog management. In: Eleventh annual conference of the international speech communication associationGoogle Scholar
  9. 9.
    Jurčíček F, Thomson B, Young S (2012) Reinforcement learning for parameter estimation in statistical spoken dialogue systems. Comput Speech Lang 26(3):168–192CrossRefGoogle Scholar
  10. 10.
    Li J, Monroe W, Ritter A, Galley M, Gao J, Jurafsky D (2016) Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:160601541
  11. 11.
    Li Z, Tang J (2017) Weakly supervised deep matrix factorization for social image understanding. IEEE Trans Image Process 26(1):276–288MathSciNetCrossRefGoogle Scholar
  12. 12.
    Li Z, Liu J, Tang J, Lu H (2015) Robust structured subspace learning for data representation. IEEE Trans Pattern Anal Mach Intell 37(10):2085–2098CrossRefGoogle Scholar
  13. 13.
    Orozko OR, Torres MI (2015) Online learning of stocastic bi-automaton to model dialogues. In: Iberian conference on pattern recognition and image analysis, Springer, Berlin, pp 441–451Google Scholar
  14. 14.
    Raux A, Langner B, Bohus D, Black AW, Eskenazi M (2005) Let’s go public! Taking a spoken dialog system to the real world. In: Ninth European conference on speech communication and technologyGoogle Scholar
  15. 15.
    Schatzmann J, Young S (2009) The hidden agenda user simulation model. IEEE Trans Audio Speech Lang Process 17(4):733–747CrossRefGoogle Scholar
  16. 16.
    Schatzmann J, Georgila K, Young S (2005) Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: 6th SIGDIAL workshop on discourse and dialogueGoogle Scholar
  17. 17.
    Schatzmann J, Weilhammer K, Stuttle M, Young S (2006) A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. Knowl Eng Rev 21(2):97–126CrossRefGoogle Scholar
  18. 18.
    Serban IV, Sordoni A, Bengio Y, Courville AC, Pineau J (2016) Building end-to-end dialogue systems using generative hierarchical neural network models. AAAI 16:3776–3784Google Scholar
  19. 19.
    Serras M, Torres MI, Del Pozo A (2017a) Online learning of attributed bi-automata for dialogue management in spoken dialogue systems. In: Iberian conference on pattern recognition and image analysis, Springer, Berlin, pp 22–31Google Scholar
  20. 20.
    Serras M, Torres MI, del Pozo A (2017b) Regularized neural user model for goal oriented spoken dialogue systems. In: International workshop on spoken dialogue systems (IWSDS)Google Scholar
  21. 21.
    Sutton RS, Barto AG (1998) Reinforcement learning: an introduction, vol 1. MIT Press, CambridgeGoogle Scholar
  22. 22.
    Thomson B, Yu K, Keizer S, Gašić M, Jurčíček F, Mairesse F, Young S (2010) Bayesian dialogue system for the let’s go spoken dialogue challenge. In: Spoken language technology workshop (SLT), 2010 IEEE, IEEE, pp 460–465Google Scholar
  23. 23.
    Torres MI (2013) Stochastic bi-languages to model dialogs. In: Proceedings of the 11th international conference on finite state methods and natural language processing, pp 9–17Google Scholar
  24. 24.
    Torres MI, Benedí JM, Justo R, Ghigi F (2012) Modeling spoken dialog systems under the interactive pattern recognition framework. In: Joint IAPR international workshops on statistical techniques in pattern recognition (SPR) and structural and syntactic pattern recognition (SSPR), Springer, Berlin, pp 519–528Google Scholar
  25. 25.
    Vinyals O, Le Q (2015) A neural conversational model. arXiv preprint arXiv:150605869
  26. 26.
    Ward W (1995) The CMU atis system. In: Proceedings of ARPA spoken language systems technology workshopGoogle Scholar
  27. 27.
    Williams JD, Zweig G (2016) End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. arXiv preprint arXiv:160601269
  28. 28.
    Young S, Gašić M, Thomson B, Williams JD (2013) Pomdp-based statistical spoken dialog systems: a review. Proc IEEE 101(5):1160–1179CrossRefGoogle Scholar
  29. 29.
    Zhao T, Eskenazi M (2016) Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. In: 17th annual meeting of the special interest group on discourse and dialogue, p 1Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Speech and Natural Language Technologies, VicomtechDonostia/San SebastiánSpain
  2. 2.Speech Interactive Research GroupUniversidad del País Vasco, UPV/EHULeioaSpain

Personalised recommendations