Abstract
One challenge for dialogue agents is to recognize the feelings of the conversation partner and respond accordingly. In this work, RoBERTa-GPT2 is proposed for empathetic dialogue generation, where the pre-trained auto-encoding RoBERTa is utilized as encoder and the pre-trained auto-regressive GPT-2 as decoder. With the combination of the pre-trained RoBERTa and GPT-2, our model realizes a new state-of-the-art emotion accuracy. To enable the empathetic ability of RoBERTa-GPT2 model, we propose a commonsense knowledge and emotional concepts extractor, in which the commonsensible and emotional concepts of dialogue context are extracted for the GPT-2 decoder. The experiment results demonstrate that the empathetic dialogue generation benefits from both pre-trained encoder-decoder architecture and external knowledge.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
References
Antoun W, Baly F, Hajj H (2020) Arabert: transformer-based model for Arabic language understanding. In: LREC 2020 workshop language resources and evaluation conference 11–16 May 2020, p 9
Budzianowski P, Vulić I (2019) Hello, it’s GPT-2-how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. In: Proceedings of the 3rd workshop on neural generation and translation, pp 15–22
Colombo P, Witon W, Modi A, Kennedy J, Kapadia M (2019) Affect-driven dialog generation. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies, vol 1 (Long and Short Papers), pp 3734–3743
Devlin J, Chang MW, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies, vol 1 (Long and Short Papers), pp 4171–4186
Dinan E, Logacheva V, Malykh V, Miller A, Shuster K, Urbanek J, Kiela D, Szlam A, Serban I, Lowe R et al (2019) The second conversational intelligence challenge (convai2), p 1902
Fan A, Lewis M, Dauphin Y (2018) Hierarchical neural story generation. In: ACL (1)
Grootendorst M (2020) Keybert: minimal keyword extraction with Bert. https://doi.org/10.5281/zenodo.4461265
Holtzman A, Buys J, Du L, Forbes M, Choi Y (2019) The curious case of neural text degeneration. In: International conference on learning representations
Hsu CC, Chen SY, Kuo CC, Huang TH, Ku LW (2018) Emotionlines: an emotion corpus of multi-party conversations. In: Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018)
Li J, Galley M, Brockett C, Gao J, Dolan WB (2016) A diversity-promoting objective function for neural conversation models. In: Proceedings of the 2016 conference of the north American chapter of the association for computational linguistics: human language technologies, pp 110–119
Li J, Galley M, Brockett C, Spithourakis G, Gao J, Dolan WB (2016) A persona-based neural conversation model. In: Proceedings of the 54th annual meeting of the association for computational linguistics (vol 1: Long Papers), pp 994–1003
Li J, Sun X (2018) A syntactically constrained bidirectional-asynchronous approach for emotional conversation generation. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 678–683
Li Q, Chen H, Ren Z, Ren P, Tu Z, Chen Z (2020) EMPDG: multi-resolution interactive empathetic dialogue generation. In: Proceedings of the 28th international conference on computational linguistics, pp 4454–4466
Li Q, Li P, Chen Z, Ren Z (2020) Towards empathetic dialogue generation over multi-type knowledge
Li Y, Su H, Shen X, Li W, Cao Z, Niu S (2017) Dailydialog: a manually labelled multi-turn dialogue dataset. In: Proceedings of the eighth international joint conference on natural language processing (vol 1: Long Papers), pp 986–995
Lin Z, Madotto A, Shin J, Xu P, Fung P (2019) MoEL: Mixture of empathetic listeners. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 121–132
Lin Z, Xu P, Winata GI, Siddique FB, Liu Z, Shin J, Fung P (2020) CAiRE: an end-to-end empathetic chatbot. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 13,622–13,623
Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized BERT pretraining approach. CoRR. arXiv:1907.11692
Loper E, Bird S (2002) NLTK: the natural language toolkit. In: Proceedings of the ACL-02 workshop on effective tools and methodologies for teaching natural language processing and computational linguistics, pp 63–70
Mohammad S (2018) Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In: Proceedings of the 56th annual meeting of the association for computational linguistics (vol 1: Long Papers), pp 174–184
Naous T, Antoun W, Mahmoud R, Hajj H (2021) Empathetic bert2bert conversational model: learning Arabic language generation with little data. In: Proceedings of the sixth arabic natural language processing workshop, pp 164–172
Peng B, Zhu C, Li C, Li X, Li J, Zeng M, Gao J (2020) Few-shot natural language generation for task-oriented dialog. In: Proceedings of the 2020 conference on empirical methods in natural language processing: findings, pp 172–182
Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543
Radford A, Narasimhan K, Salimans T, Sutskever I Improving language understanding by generative pre-training
Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9
Rashkin H, Smith EM, Li M, Boureau YL (2018) I know the feeling: learning to converse with empathy
Rashkin H, Smith EM, Li M, Boureau YL (2019) Towards empathetic open-domain conversation models: a new benchmark and dataset. In: ACL (1)
Rothe S, Narayan S, Severyn A (2020) Leveraging pre-trained checkpoints for sequence generation tasks. Trans Assoc Comput Linguist 8:264–280
Sennrich R, Haddow B, Birch A (2016) Neural machine translation of rare words with subword units. In: Proceedings of the 54th annual meeting of the association for computational linguistics (vol 1: Long Papers), pp 1715–1725
Serban IV, Lowe R, Charlin L, Pineau J (2016) Generative deep neural networks for dialogue: a short review
Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J (2015) Hierarchical neural network generative models for movie dialogues 7(8):434–441. arXiv:1507.04808
Shen L, Feng Y (2020) CDL: curriculum dual learning for emotion-controllable response generation. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 556–566
Skowron M, Theunis M, Rank S, Kappas A (2013) Affect and social processes in online communication-experiments with an affective dialog system. IEEE Trans Affect Comput 4(3):267–279
Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Proceedings of the AAAI conference on artificial intelligence, vol 31
Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. Adv Neural Inf Process Syst 27:3104–3112
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: NIPS
Wei W, Liu J, Mao X, Guo G, Zhu F, Zhou P, Hu Y (2019) Emotion-aware chat machine: automatic emotional response generation for human-like emotional interaction. In: Proceedings of the 28th ACM international conference on information and knowledge management, pp 1401–1410
Wolf T, Sanh V, Chaumond J, Delangue C (2019) Transfertransfo: a transfer learning approach for neural network based conversational agents
Young T, Cambria E, Chaturvedi I, Zhou H, Biswas S, Huang M (2018) Augmenting end-to-end dialogue systems with commonsense knowledge. In: Proceedings of the AAAI conference on artificial intelligence, vol 32
Zandie R, Mahoor MH (2020) Emptransfo: a multi-head transformer architecture for creating empathetic dialog systems. In: The thirty-third international flairs conference
Zech E, Rimé B (2005) Is talking about an emotional experience helpful? effects on emotional recovery and perceived benefits. Clin Psychol Psychother: Int J Theory Pract 12(4):270–287
Zhang S, Dinan E, Urbanek J, Szlam A, Kiela D, Weston J (2018) Personalizing dialogue agents: i have a dog, do you have pets too? In: ACL (1)
Zhang Y, Sun S, Galley M, Chen YC, Brockett C, Gao X, Gao J, Liu J, Dolan WB (2020) Dialogpt: large-scale generative pre-training for conversational response generation. In: Proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations, pp 270–278
Zhong P, Wang D, Miao C (2019) Knowledge-enriched transformer for emotion detection in textual conversations. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 165–176
Zhou H, Huang M, Zhang T, Zhu X, Liu B (2018) Emotional chatting machine: emotional conversation generation with internal and external memory. In: Thirty-second AAAI conference on artificial intelligence
Zhou X, Wang WY (2018) Mojitalk: generating emotional responses at scale. In: Proceedings of the 56th annual meeting of the association for computational linguistics (vol 1: Long Papers), pp 1128–1137
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, Y., Maier, W., Minker, W., Ultes, S. (2022). Empathetic Dialogue Generation with Pre-trained RoBERTa-GPT2 and External Knowledge. In: Stoyanchev, S., Ultes, S., Li, H. (eds) Conversational AI for Natural Human-Centric Interaction. Lecture Notes in Electrical Engineering, vol 943. Springer, Singapore. https://doi.org/10.1007/978-981-19-5538-9_5
Download citation
DOI: https://doi.org/10.1007/978-981-19-5538-9_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-5537-2
Online ISBN: 978-981-19-5538-9
eBook Packages: Computer ScienceComputer Science (R0)