Abstract
To promote the research and development of dialogue breakdown detection for dialogue systems, we have been organizing a series of dialogue breakdown detection challenges to detect a system’s inappropriate utterances that lead to dialogue breakdowns in chat-oriented dialogue. In this paper, we overview Dialogue Breakdown Detection Challenge 4 (DBDC4). As in the previous challenges, we used datasets in English and Japanese. Four teams participated in the challenge, in which all four teams worked on English, and two of the four teams worked on Japanese as well. This paper describes the task setting, evaluation metrics, and datasets for the challenge and the results of the submitted runs of the participants.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Banchs RE, Li H (2012) IRIS: a chat-oriented dialogue system based on the vector space model. In: Proceedings of the ACL 2012 system demonstrations, pp 37–42
Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805
D’Haro LF, Shawar BA, Yu Z (2016) RE-WOCHAT 2016—shared task description report. In: Proceedings of the workshop on collecting and generating resources for chatbots and conversational agents-development and evaluation (RE-WOCHAT), pp 39–42
Hendriksen M, Leeuwenberg A, Moens MF (2019) LSTM for dialogue breakdown detection: exploration of different model types and word embeddings. In: Proceedings of the WOCHAT
Higashinaka R, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y, Mizukami M (2015) Towards taxonomy of errors in chat-oriented dialogue systems. In: Proceedings of the SIGDIAL, pp 87–95
Higashinaka R, Mizukami M, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y (2015) Fatal or not? Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems. In: Proceedings of the EMNLP, pp 2243–2248
Higashinaka R, Funakoshi K, Kobayashi Y, Inaba M (2016) The dialogue breakdown detection challenge: task description, datasets, and evaluation metrics. In: Proceedings of the LREC, pp 3146–3150
Higashinaka R, Funakoshi K, Inaba M, Tsunomori Y, Takahashi T, Akama R (2019) Dialogue system live competition: identifying problems with dialogue systems through live event. In: Proceedings of the IWSDS
Martinovsky B, Traum D (2003) The error is the clue: breakdown in human-machine interaction. In: Proceedings of the ISCA workshop on error handling in spoken dialogue systems, pp 11–16
Onishi K, Yoshimura T (2014) Casual conversation technology achieving natural dialog with computers. NTT DOCOMO Tech J 15(4):16–21
Ritter A, Cherry C, Dolan WB (2011) Data-driven response generation in social media. In: Proceedings of the EMNLP, pp 583–593
Shin J, Dirafzoon A, Anshu A (2019) Context-enriched attentive memory network with global and local encoding for dialogue breakdown detection. In: Proceedings of the WOCHAT
Sugiyama H (2019) Dialogue breakdown detection using BERT with traditional dialogue features. In: Proceedings of the WOCHAT
Tsukahara H, Uchiumi K (2015) System utterance generation by label propagation over association graph of words and utterance patterns for open-domain dialogue systems. In: Proceedings of the PACLIC, pp 323–331
Tsunomori Y, Higashinaka R, Takahashi T, Inaba M (2018) Evaluating dialogue breakdown detection in chat-oriented dialogue systems. In: Proceedings of the SEMDIAL
Wang C, Kato S, Sakai T (2019) RSL19BD at DBDC4: ensemble of decision tree-based and LSTM-based models. In: Proceedings of the WOCHAT
Acknowledgements
We would like to thank the participants for their efforts to explore new methods and for submitting their runs and system description papers. We thank Rafael E. Banchs and members of the Conversational Intelligence Challenge for graciously providing us with datasets to make DBDC4 possible. We also thank our sponsors, Denso IT Laboratories and National University of Singapore, for supporting our data collection.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Higashinaka, R. et al. (2021). Overview of the Dialogue Breakdown Detection Challenge 4. In: Marchi, E., Siniscalchi, S.M., Cumani, S., Salerno, V.M., Li, H. (eds) Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Lecture Notes in Electrical Engineering, vol 714. Springer, Singapore. https://doi.org/10.1007/978-981-15-9323-9_38
Download citation
DOI: https://doi.org/10.1007/978-981-15-9323-9_38
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9322-2
Online ISBN: 978-981-15-9323-9
eBook Packages: EngineeringEngineering (R0)