Skip to main content
SpringerLink
Log in

Inferring Stressors from Conversation: Towards an Emotional Support Robot Companion

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

People build relationships with each other through emotional support. Appropriate care provided by friends or family may critically help people extricate themselves from negative emotional states. In this work, we implement a conversational robot that can provide emotional support to people facing stressful situations in their daily lives. The proposed robot system extracts meaningful keywords from the support recipients’ speech to identify the moments when people expose their concerns through self-disclosure. These keywords are then used to determine the underlying stressors in the recipients’ minds. By using a novel approach through a commonsense knowledge graph-based Bayesian network, the proposed system leverages Gibbs sampling to infer the posterior probability of various documented stressors. Moreover, we trained a neural network to determine an appropriate support strategy that should be offered to a recipient with a specific personality given a detected stressor. We evaluated our implementation through several human-in-loop experiments. Results show that this system can correctly identify the underlying stressors from speech and provide appropriate emotional support required to help people cope with difficult situations in one’s life.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Availability of data and material

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Asher SR, Parker JG, Walker DL (1996) Distinguishing friendship from acceptance: Implications for intervention and assessment. The company they keep: Friendships in childhood adolescence, 366–405

  2. Cohen S, Wills TA (1985) “Stress, social support, and the buffering hypothesis,” Psychological bulletin, 310

  3. Shavitt S, Cho YI, Johnson TP, Jiang D, Holbrook A, Stavrakantonaki M (2016) Culture moderates the relation between perceived stress, social support, and mental and physical health. Journal of Cross-Cultural Psychology 47:956–980

    Article  Google Scholar 

  4. Lehman DR, Hemphill KJ (1990) Recipients’ perceptions of support attempts and attributions for support attempts that fail. Journal of Social and Personal Relationships 7:563–574

    Article  Google Scholar 

  5. Heerink M, Krose B, Evers V, Wielinga B (2006) “The influence of a robot’s social abilities on acceptance by elderly users.” In: IEEE Int. Symposium on Robot and Human Interactive Communication (RO-MAN), UK, 521–526

  6. Samter W, Burleson BR (1990) Evaluations of communication skills as predictors of peer acceptance in a group living situation. Communication Studies 41:311–326

    Article  Google Scholar 

  7. Burleson BR (1994) Comforting messages: Features, functions, and outcomes. Strategic interpersonal communication, 135-161

  8. Ickes WJ (Ed.) Empathic accuracy. The Guilford Press, (1997)

  9. Barker RL (2008) The social work dictionary. NASW Press, Washington, DC

    Google Scholar 

  10. Marsella S, Gratch J, Petta P (2010) Computational models of emotion. In: Scherer KR, Bänziger T, Roesch E (eds) A blueprint for an affectively competent agent: Cross-fertilization between Emotion Psychology, Affective Neuroscience, and Affective Computing. Oxford University Press, Oxford, pp 21–46

    Google Scholar 

  11. Gilliland AL (2011) After praise and encouragement: Emotional support strategies used by birth doulas in the USA and Canada. Midwifery 27:525–531

    Article  Google Scholar 

  12. Baylor AL, Warren D, Park S, Shen E, Perez R (2005) The impact of frustration-mitigating messages delivered by an interface agent. Artificial intelligence in education: supporting learning through intelligent socially informed technology 125:73

    Google Scholar 

  13. Prendinger H, Ishizuka M (2005) The empathic companion: A character-based interface that addresses users’ affective states. Applied Artificial Intelligence 19:267–285

    Article  Google Scholar 

  14. Dennis M, Masthoff J (2016) Adapting progress feedback and emotional support to learner personality. International Journal of Artificial Intelligence in Education 26:877–931

    Article  Google Scholar 

  15. Cuff BMP, Brown SJ, Taylor L, Howat DJ (2016) Empathy: A Review of the Concept. Emot. Rev. 8(2):144–153. https://doi.org/10.1177/1754073914558466

    Article  Google Scholar 

  16. Cutrona CE, Cohen BB, Igram S (1990) Contextual determinants of the perceived supportiveness of helping behaviors. Journal of Social Personal Relationships 7:553–562

    Article  Google Scholar 

  17. Cobb S (1976) Social support as a moderator of life stress, Psychosomatic medicine, 300-314

  18. House J (1983) Work stress and social support

  19. Burleson BR, Samter W (1985) Consistencies in theoretical and naive evaluations of comforting messages. Communications Monographs 52:103–123

    Article  Google Scholar 

  20. Greenberg LS, Rice LN, Elliott R (1996) Facilitating emotional change: The moment-by-moment process

  21. Goldsmith DJ, MacGeorge EL (2000) The impact of politeness and relationship on perceived quality of advice about a problem. Human Communication Research 26:234–263

    Article  Google Scholar 

  22. Wada K, Shibata T (2007) Living with seal robots—its sociopsychological and physiological influences on the elderly at a care house. IEEE transactions on robotics 23:972–980

    Article  Google Scholar 

  23. Ullrich D, Diefenbach S, Butz A (2016) Murphy Miserable Robot: A Companion to Support Children’s Well-being in Emotionally Difficult Situations, CHI Conference Extended Abstracts on Human Factors in Computing Systems, 3234-3240

  24. Gongor F, Tutsoy O (2019) Design and Implementation of a Facial Character Analysis Algorithm for Humanoid Robots. Robotica. https://doi.org/10.1017/S0263574719000304

  25. Gamborino E, Fu LC (2018) Interactive Reinforcement Learning based Assistive Robot for the Emotional Support of Children. Proceedings of the IEEE 18th International Conference on Control and Automation Systems

  26. Gamborino E, Yueh H-P, Lin W, Yeh S-L, Fu L-C (2019) “Mood Estimation as Social Profile Predictor in an Autonomous, Multi-Session, Emotional Support Robot for Children”, Proceedings of the 28th IEEE International Conference on Robot and Human Interactive Communication

  27. Hoey J, Schröder T, Alhothali A (2016) Affect control processes: Intelligent affective interaction using a partially observable Markov decision process. Artificial Intelligence 230:134–172

    Article  MathSciNet  Google Scholar 

  28. Dennis M, Kindness P, Masthoff J, Mellish C, Smith K (2013) Towards effective emotional support for community first responders experiencing stress, Humaine Association Conference on Affective Computing and Intelligent Interaction, 763-768

  29. Kindness P, Masthoff J, Mellish C (2017) Designing emotional support messages tailored to stressors. International Journal of Human-Computer Studies 97:1–22

    Article  Google Scholar 

  30. Ho A, Hancock J, Miner AS (2018) Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot. Journal of Communication 68(4):712–733

    Article  Google Scholar 

  31. Cambria E, Havasi C, Hussain A (2012) SenticNet 2: A Semantic and Affective Resource for Opinion Mining and Sentiment Analysis. International Florida Artificial Intelligence Research Society Conference

  32. Mohammad SM, Turney P (2013) Crowdsourcing a word–emotion association lexicon. Computational Intelligence 29:436–465

    Article  MathSciNet  Google Scholar 

  33. Agarwal B, Mittal N, Bansal P, Garg S (2015) Sentiment analysis using common- sense and context information. Computational intelligence neuroscience, 30

  34. Wu Y-L, Gamborino E, Fu L-C (2019) Interactive Question Posing System for Robot-Assisted Photo Reminiscence from Personal Photos. IEEE Transactions on Cognitive and Developmental Systems

  35. Speer R, Chin J, Havasi C (2017) “Conceptnet 5.5: An open multilingual graph of general knowledge”. Thirty-First AAAI Conference on Artificial Intelligence

  36. Abdul-Mageed L (2017) “EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks”. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 718–728)

  37. Winata GI, Kampman OP, Fung P (2018) “Attention-Based LSTM for Psychological Stress Detection from Spoken Language Using Distant Supervision.” 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, 6204-6208, https://doi.org/10.1109/ICASSP.2018.8461990

  38. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) “Distributed Representations of Wornameds and Phrases and their Compositionality.” In: Conference on Neural Information Processing Systems, 3111–3119

  39. Hochreiter S, Schmidhuber J (1997) Long Short-Term Memory Neural Computation. Neural Computation 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  40. Zhong C (2019) “Knowledge-Enriched Transformer for Emotion Detection in Textual Conversations.” In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 165–176)

  41. Lea RG, Davis SK, Mahoney B, Qualter P (2019) Does Emotional Intelligence Buffer the Effects of Acute Stress? A Systematic Review, Frontiers in psychology 10:810. https://doi.org/10.3389/fpsyg.2019.00810

    Article  Google Scholar 

  42. Saha A, Aralikatte R, Khapra MM, Sankaranarayanan K (2019) “DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension”. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4487–4496

  43. Malmasi S, Dras M, Johnson M, Du L, Wolska M (2017) “Unsupervised Text Segmentation Based on Native Language Characteristics”. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1457–1469

  44. Kim J, Kim Y, Sarikaya R, Fosler-Lussier E (2017) “Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources”. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2832–2838

  45. Zhang M, Zhang Y, Che W, Liu T (2014) “Character-Level Chinese Dependency Parsing”. Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics, 1326-1336

  46. Freedman R (2000) “Plan-Based Dialogue Management in a Physics Tutor”, Sixth Applied Natural Language Processing Conference, Seattle, Washinton, USA, 52–59

  47. Ham D, Lee J, Jang Y, Kim K (2020) “End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2”. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 583–592

  48. Weizenbaum J (1966) “ELIZA - a computer program for the study of natural language communication between man and machine”. Communications of the acm

  49. Angara P, Jimenez M, Agarwal K, Jain H, Jain R, Stege U, Ganti S, Muller H (2017) “Foodie Fooderson, A Conversational Agent for the Smart Kitchen. In: Proceedings of the 27th Annual International Conference on Computer Science and Software Engineering

  50. Gao J, Galley M, Li L (2018) “Neural Approaches to Conversational AI”. In: Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, 1371–1374

  51. Lafferty J, McCallum A, Pereira FC (2001) “Conditional random fields: Probabilistic models for segmenting and labeling sequence data”. In: International Conference on Machine Learning, 282-289

  52. Tian Y (2020) “Improving Chinese Word Segmentation with Wordhood Memory Networks”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8274–8285)

  53. Huang W (2020) “Towards Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning.” In: Proceedings of the 28th International Conference on Computational Linguistics 2062–2072

  54. Yang S (2019) “Subword Encoding in Lattice LSTM for Chinese Word Segmentation”. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 2720–2725)

  55. Che W, Li Z, Liu T (2010) “Ltp: A chinese language technology platform”, International Conference on Computational Linguistics: Demonstrations, 13-16

  56. Hart SG (2006) “NASA-task load index (NASA-TLX); 20 years later”, human factors and ergonomics society annual meeting, 904–908

  57. Gadzella BM (1994) Student-life stress inventory: Identification of and reactions to stressors. Psychological reports 74:395–402

    Article  Google Scholar 

  58. Hamaideh SH (2011) Stressors and reactions to stressors among university students. International journal of social psychiatry 57:69–80

    Article  Google Scholar 

  59. Holmes TH, Rahe RH (1967) The social readjustment rating scale. Journal of psychosomatic research 11:213–218

    Article  Google Scholar 

  60. Ajibade B, Olabisi O, Fabiyi B, Ajao O, Ayeni A (2016) “Stress, types of stressors and coping strategies amongst selected nursing schools students in South-West, Nigeria”. European Journal of Biology Medical Science Research, 1-15

  61. Daily Inventory of Stressful Events, [Online]. https://scienceofbehaviorchange.org/

  62. ISMA Stress Questionnaire, [Online]. https://isma.org.uk/wp-content/uploads/2013/08/Stress-Questionnaire.pdf

  63. UCU model stress questionnaire, [Online]. https://www.ucu.org.uk/stress

  64. Guo H, Hsu W (2002) “A survey of algorithms for real-time Bayesian network inference.” In: Joint Workshop on Real Time Decision Support and Diagnosis Systems

  65. Kopacz M (2005) “Personality and music preferences: The influence of personality traits on preferences regarding musical elements”. Journal of Music Therapy, pp.216-239

  66. R. C. Mulyanegara, Y. Tsarenko, and A. Anderson, The Big Five and brand personality: Investigating the impact of consumer personality on preferences towards particular brand personality, Journal of Brand Management, pp.234-247, 2009

  67. Chan DW (1994) The Chinese Ways of Coping Questionnaire: Assessing coping in secondary school teachers and students in Hong Kong. Psychological Assessment 6(2):108–116

    Article  Google Scholar 

  68. Worswick S, Mitsuku, [Online]. https://www.pandorabots.com/mitsuku/

  69. Thompson E (2008) Development and validation of an international English big-five mini-markers. Personality individual differences 45:542–548

    Article  Google Scholar 

  70. Deng J, Zeng X, Li Y, You C (2011) International English Big-Five Mini Markers in , 579-615

  71. Dice LR (1945) Measures of the amount of ecologic association between species, Ecology, 297-302

Download references

Funding

This research was supported by the Ministry of Science and Technology of Taiwan, and Center for Artificial Intelligence & Advanced Robotics, National Taiwan University, under the grant numbers MOST 110-2634-F-002-049 & MOST 110-2221-E-002-166-MY3.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan

    Yu-Cian Huang & Yan-Jia Huang

  2. Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan

    Xiaobei Qian

  3. Research Center for Artificial Intelligence and Advanced Robotics, National Taiwan University, Taipei, Taiwan

    Edwinn Gamborino & Li-Chen Fu

  4. Department of Psychology, National Taiwan University, Taipei, Taiwan

    Su-Ling Yeh

Authors
  1. Yu-Cian Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Edwinn Gamborino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan-Jia Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaobei Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Li-Chen Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Su-Ling Yeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li-Chen Fu.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Consent to participate

All participants signed a consent form for the use and storage of their personal data for research purposes, both the submitted files and any audiovisual recordings of the experiment sessions.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, YC., Gamborino, E., Huang, YJ. et al. Inferring Stressors from Conversation: Towards an Emotional Support Robot Companion. Int J of Soc Robotics 14, 1657–1671 (2022). https://doi.org/10.1007/s12369-022-00902-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-022-00902-0

Keywords

Search

Navigation