Abstract
Spoken dialogue system interfaces are gaining increasing attention, with examples including Apple’s Siri, Amazon’s Alexa, and numerous other products. Yet most existing solutions remain heavily data-driven, and face limitations in integrating and handling data semantics. They mainly rely on statistical co-occurrences in the training dataset and lack a more profound knowledge integration model with semantically structured information such as knowledge graphs. This paper evaluates the impact of performing knowledge base integration (KBI) to regulate the dialogue output of a deep learning conversational system. More specifically, it evaluates whether integrating dependencies between the data, obtained from the semantic linking of an external knowledge base (KB), would help improve conversational quality. To do so, we compare three approaches of conversation preprocessing methods: i) no KBI: considering conversational data with no external knowledge integration, ii) All Predicates KBI: considering conversational data where all dialogue pairs are augmented with their linked predicates from the domain KB, and iii) Intersecting Predicates KBI: considering conversational data where dialogue pairs are augmented only with their intersecting predicates (to filter-out potentially useless or redundant knowledge). We vary the amount of history considered in the conversational data, ranging from 0% (considering the last dialogue pair only) to 100% (considering all dialogue pairs, from the beginning of the dialogue). To our knowledge, this is the first study to evaluate knowledge integration in the preprocessing phase of conversational systems. Results are promising and show that knowledge integration – with an amount of history ranging between 10% and 75%, generally improves conversational quality.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
It is called one-hot because only one bit is “hot” or TRUE at any time. For example, a 3-bit one-hot encoding would have three states: 001, 010, and 100, compared with 23 binary combinations obtained with binary encoding. Note that other encodings such as word2vec and GloVe vetor representations can be used.
- 2.
Available online at: https://github.com/HLTCHKUST/Mem2Seq/tree/master/data/dialog-bAbI-tasks
- 3.
References
Ahn, S., et al.: A Neural Knowledge Language Model (2016). CoRR abs/1608.00318
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: International Conference on Learning Representations (ICLR) (2015)
Bordes, A., Boureau, Y., Weston, J.: Learning end-to-end goal-oriented dialog. In: International Conference on Learning Representations (ICLR) (2017)
Collier, M., Beel, J.: Implementing neural turing machines. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 94–104. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01424-7_10
Deng, J., et al.: Large-scale object classification using label relation graphs. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 48–64. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_4
Deriu, J., et al.: Survey on Evaluation Methods for Dialogue Systems (2019). CoRR abs/1905.04071
Haraty, R., El Ariss, O.: Lebanese colloquial arabic speech recognition. In: ISCA International Conference on Computer Applications in Industry and Engineering (CAINE), pp. 285–291 (2005)
Haraty, R., Nasrallah, R.: Indexing arabic texts using association rule data mining. Libr. Hi Tech 37(1), 101–117 (2019)
Henderson, M., Williams, J., Thomson, B.: The second dialog state tracking challenge. In: SIGDIAL Conference, pp. 263–272 (2014)
Ticona-Herrera, R., Tekli, J., Chbeir, R., Laborie, S., Dongo, I., Guzman, R.: Toward RDF normalization. In: Johannesson, P., Lee, M.L., Liddle, S.W., Opdahl, A.L., López, Ó.P. (eds.) Conceptual Modeling, pp. 261–275. Springer International Publishing, Cham (2015). https://doi.org/10.1007/978-3-319-25264-3_19
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hu, Z., et al.: Harnessing deep neural networks with logic rules. In: Annual Meeting of the Association for Computational Linguistics (ACL) (2016)
Hu, Z., et al.: Deep neural networks with massive learned knowledge. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1670–1679 (2016)
Jafari, R., Razvarz, S., Gegov, A.: End-to-end memory networks: a survey. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) SAI 2020. AISC, vol. 1229, pp. 291–300. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52246-9_20
Kalchbrenner, N., Blunsom, P.: Recurrent continuous translation models. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1700–1709 (2013)
Karaletsos, T., Belongie, S.J., Rätsch, G.: When crowds hold privileges: bayesian unsupervised representation learning with oracle constraints. In: International Conference on Learning Representations (ICLR) (2016)
Li, J., et al.: A persona-based neural conversation model. In: Annual Meeting of the Association for Computational Linguistics (ACL) (2016)
Liu, Y., et al.: RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019). CoRR abs/1907.11692
Miller, G.A., Fellbaum, C.: WordNet then and now. Lang. Resour. Eval. 41(2), 209–214 (2007)
Nguyen, A., Yosinski, J., Clune J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 427–436 (2015)
Qu, C., et al.: BERT with history answer embedding for conversational question answering. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1133–1136 (2019)
Ritter, A., Cherry, C., Dolan, W.: Data-driven response generation in social media. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 583–593 (2011)
Serban, J., Lowe, R., et al.: A survey of available corpora for building data-driven dialogue systems: the journal version. Dial. Discourse 9(1), 1–49 (2018)
Shaik, R., et al.: The analysis of data representation techniques for early prediction of breast cancer. Int. J. Pure Appl. Math., 1311–8080 (2017)
Shang, L., Lu, Z., Li, H.: Neural responding machine for short-text conversation. In: Annual Meeting of the Association for Computer Linguistics (ACL), pp. 1577–1586 (2015)
Sordoni, A., et al.: A neural network approach to context-sensitive generation of conversational responses. In: North American Chapter of the Association for Computational Linguistics (NAACL), pp. 196–205 (2015)
Sukhbaatar, S., et al.: End-to-end memory networks. In: Neural Information Processing Systems (NeurIPS), pp. 2440–2448 (2015)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Neural Information Processing Systems (NeurIPS), pp. 3104–3112 (2014)
Szegedy, C., et al.: Intriguing properties of neural networks. In: International Conference on Learning Representations (ICLR) (2013)
Vinyals, O., et al.: Grammar as a foreign language. In: Neural Information Processing Systems (NeurIPS), pp. 2773–2781 (2015)
Vinyals, O., Le, Q.: A Neural Conversational Model (2015). CoRR abs/1506.05869
Vinyals, O., et al.: Show and tell: a neural image caption generator. In: Computer Vision and Pattern Recognition (CVPR), pp. 3156–3164 (2015)
Wen, T., et al.: A network-based end-to-end trainable task-oriented dialogue system. In: Conference of the European Chapter of the Association for Computational Linguistics (EACL), pp. 438–449 (2017)
Williams, J., Raux, A., Henderson, M.: The dialog state tracking challenge series: a review. Dial. Discourse 7(3), 4–33 (2016)
Yang, Z., et al.: XLNET: generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, pp. 5753–5763 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Maziad, H., Rammouz, JA., Asmar, B.E., Tekli, J. (2021). Preprocessing Techniques for End-To-End Trainable RNN-Based Conversational System. In: Brambilla, M., Chbeir, R., Frasincar, F., Manolescu, I. (eds) Web Engineering. ICWE 2021. Lecture Notes in Computer Science(), vol 12706. Springer, Cham. https://doi.org/10.1007/978-3-030-74296-6_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-74296-6_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74295-9
Online ISBN: 978-3-030-74296-6
eBook Packages: Computer ScienceComputer Science (R0)