Skip to main content

Simulated Task Oriented Dialogues for Developing Versatile Conversational Agents

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14608))

Included in the following conference series:

Abstract

Task-Oriented Dialogue (TOD) Systems are increasingly important for managing a variety of daily tasks, yet often underperform in unfamiliar scenarios due to limitations in existing training datasets. This study addresses the challenge of generating robust and versatile TOD systems by transforming instructional task descriptions into natural user-system dialogues to serve as enhanced pre-training data. We explore three strategies for synthetic dialogue generation: crowdsourcing, encoder-decoder models, and in-context learning with large language models. The evaluation of these approaches, based on a comprehensive user study employing 10 different metrics, reveals the top quality of the dialogues generated by learning an encoder-decoder model as per human evaluation. Notably, employing this synthetic dialogue further improves the performance of advanced TOD models, especially in unfamiliar domains, with improvements spanning 5.5% to as much as 20.9% in combined evaluation scores. Our findings advocate for the use of specialised, task-oriented knowledge bases and step-wise dialogue generation techniques to advance the capabilities and generalizability of TOD systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://wikihow.com.

  2. 2.

    https://github.com/wangxieric/task2kb-resource.

  3. 3.

    https://www.mturk.com/.

  4. 4.

    https://en.wikipedia.org/wiki/English-speaking_world.

References

  1. Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., Chappidi, S.: Open-domain question answering goes conversational via question rewriting. In: Proceedings of NAACL (2021)

    Google Scholar 

  2. Bao, J., et al.: A synthetic data generation framework for grounded dialogues. In: Proceedings of ACL (2023)

    Google Scholar 

  3. Boyer, K., Ha, E.Y., Phillips, R., Wallis, M., Vouk, M., Lester, J.: Dialogue act modeling in a complex task-oriented domain. In: Proceedings of SIGDIAL (2010)

    Google Scholar 

  4. Budzianowski, P., et al.: MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In: Proceedings of EMNLP (2018)

    Google Scholar 

  5. Cai, Y., Liu, H., Ou, Z., Huang, Y., Feng, J.: Advancing semi-supervised task oriented dialog systems by JSA learning of discrete latent variable models. In: Proceedings of SIGDIAL (2022)

    Google Scholar 

  6. Chen, D., Yu, Z.: Sources of noise in dialogue and how to deal with them. In: Proceedings of SIGDIAL (2023)

    Google Scholar 

  7. Chen, X., Xu, J., Xu, B.: A working memory model for task-oriented dialog response generation. In: Proceedings of ACL (2019)

    Google Scholar 

  8. Chung, H.W., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022)

  9. Dai, Z., et al.: Dialog inpainting: turning documents into dialogs. In: Proceedings of ICML (2022)

    Google Scholar 

  10. De Cicco, R., Silva, S.C.L.d.C.e., Alparone, F.R.: “It’s on its way": chatbots applied for online food delivery services, social or task-oriented interaction style? J. Foodserv. Bus. Res. 24(2), 140–164 (2021)

    Google Scholar 

  11. El Asri, L., et al.: Frames: a corpus for adding memory to goal-oriented dialogue systems. In: Proceedings of SIGdial (2017)

    Google Scholar 

  12. Eric, M., et al.: Multiwoz 2.1: a consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In: Proceedings of LREC (2020)

    Google Scholar 

  13. Hosseini-Asl, E., McCann, B., Wu, C.S., Yavuz, S., Socher, R.: A simple language model for task-oriented dialogue. In: Proceedings of NeurIPS (2020)

    Google Scholar 

  14. Hosseini-Asl, E., McCann, B., Wu, C., Yavuz, S., Socher, R.: A simple language model for task-oriented dialogue. In: Proceedings of NeurIPS (2020)

    Google Scholar 

  15. Hu, W., et al.: Overcoming catastrophic forgetting for continual learning via model adaptation. In: Proceeding of ICLR (2019)

    Google Scholar 

  16. Jin, D., Kim, S., Hakkani-Tur, D.: Can i be of further assistance? using unstructured knowledge access to improve task-oriented conversational modeling. In: Proceedings of DialDoc (2021)

    Google Scholar 

  17. Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: Proceedings of ICLR (2015)

    Google Scholar 

  18. Li, J.J., Nenkova, A.: Fast and accurate prediction of sentence specificity. In: Proceedings of AAAI (2015)

    Google Scholar 

  19. Madotto, A., Wu, C.S., Fung, P.: Mem2seq: effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In: Proceedings of ACL (2018)

    Google Scholar 

  20. Mohapatra, B., Pandey, G., Contractor, D., Joshi, S.: Simulated chats for building dialog systems: learning to generate conversations from instructions. In: Proceedings of EMNLP (2021)

    Google Scholar 

  21. OpenAI: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

  22. Ou, Z., Song, Y.: Joint stochastic approximation and its application to learning discrete latent variable models. In: Proceedings of UAI (2020)

    Google Scholar 

  23. Papangelis, A., Wang, Y.C., Molino, P., Tur, G.: Collaborative multi-agent dialogue model training via reinforcement learning. In: Proceedings of SIGDIAL (2019)

    Google Scholar 

  24. Peng, B., Li, C., Li, J., Shayandeh, S., Liden, L., Gao, J.: SOLOIST: building task bots at scale with transfer learning and machine teaching. Trans. Assoc. Comput. Linguist. 9, 824–907 (2021)

    Google Scholar 

  25. Procheta, S., Xi, W., Ruiqing, X., Emine, Y.: Task2kb: a public task-oriented knowledge base. In: Proceedings of AAAI (2023)

    Google Scholar 

  26. Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W.B., Iyyer, M.: Open-retrieval conversational question answering. In: Proceedings of SIGIR (2020)

    Google Scholar 

  27. Quan, J., Zhang, S., Cao, Q., Li, Z., Xiong, D.: Risawoz: a large-scale multi-domain wizard-of-oz dataset with rich semantic annotations for task-oriented dialogue modeling. In: Proceedings of EMNLP (2020)

    Google Scholar 

  28. Radford, A., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)

    Google Scholar 

  29. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)

    Google Scholar 

  30. Rastogi, A., Zang, X., Sunkara, S., Gupta, R., Khaitan, P.: Towards scalable multi-domain conversational agents: the schema-guided dialogue dataset. In: Proceedings of AAAI (2020)

    Google Scholar 

  31. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)

  32. See, A., Roller, S., Kiela, D., Weston, J.: What makes a good conversation? how controllable attributes affect human judgments. In: Proceedings of NAACL-HLT (2019)

    Google Scholar 

  33. Sen, P., Wang, X., Xu, R., Yilmaz, E.: Task2kb: a public task-oriented knowledge base. In: Proceedings of AAAI (2023)

    Google Scholar 

  34. Shah, P., et al.: Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871 (2018)

  35. Shuster, K., Poff, S., Chen, M., Kiela, D., Weston, J.: Retrieval augmentation reduces hallucination in conversation. In: Proceedings of EMNLP (Findings) (2021)

    Google Scholar 

  36. Srivastava, M., Lu, Y., Peschon, R., Li, C.: Pretrain-finetune based training of task-oriented dialogue systems in a real-world setting. In: Proceedings of NAACL (2021)

    Google Scholar 

  37. Wei, J., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of NeurIPS (2022)

    Google Scholar 

  38. Yang, Y., Li, Y., Quan, X.: Ubar: towards fully end-to-end task-oriented dialog system with gpt-2. In: Proceedings of AAAI (2021)

    Google Scholar 

  39. Ye, F., Wang, X., Huang, J., Li, S., Stern, S., Yilmaz, E.: Metaassist: robust dialogue state tracking with meta learning. In: Proceedings of EMNLP (2022)

    Google Scholar 

  40. Zang, X., Rastogi, A., Sunkara, S., Gupta, R., Zhang, J., Chen, J.: Multiwoz 2.2 : a dialogue dataset with additional annotation corrections and state tracking baselines. arXiv preprint arXiv:2007.12720 (2020)

Download references

Acknowledgement

This research is supported by the Alan Turing Institute under the EPSRC grant [EP/N510129/1] and the EPSRC Fellowship titled “Task Based Information Retrieval" [EP/P024289/1].

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xi Wang .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, X., Sen, P., Li, R., Yilmaz, E. (2024). Simulated Task Oriented Dialogues for Developing Versatile Conversational Agents. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14608. Springer, Cham. https://doi.org/10.1007/978-3-031-56027-9_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56027-9_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56026-2

  • Online ISBN: 978-3-031-56027-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics