Skip to main content
SpringerLink
Log in

Affect Sensing Using Linguistic, Semantic and Cognitive Cues in Multi-threaded Improvisational Dialogue

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

In this paper, a context-based affect detection component embedded in an improvisational virtual platform is implemented. The software allows up to five human characters and one intelligent agent to be engaged in one session to conduct creative improvisation within loose scenarios. The transcripts produced showed several conversations being conducted in parallel. Some of these conversations reveal personal subjective opinions or feelings about situations, while others are caused by social interactions and show opinions and emotional responses to other participant characters. These two types of conversations serve to inform the descriptions of the personal and the social contexts, respectively. In order to detect affect from such contexts, first of all a naïve Bayes classifier is used to categorize these two types of conversations based on linguistic cues. A semantic-based analysis is also used to further derive the discussion themes and identify the target audiences for the social interaction inputs. Then, two statistical approaches have been developed to provide affect detection, respectively, in the social and personal emotion contexts. The emotional history of each individual character is used in interpreting affect relating to the personal contexts, while the social context affect detection takes account of interpersonal relationships, sentence types, emotions implied by the potential target audiences in their most recent interactions and discussion themes. The new development of context-based affect detection is integrated with the intelligent agent. The work addresses one challenging cognitive topic in the affective computing field, the detection and revealing of the relevant “context” to inform affect detection. The work addresses the journal’s themes on human emotion behavior analysis and understanding.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. The Crohn’s disease scenario is mainly about Peter who has had Crohn’s disease since the age of 15. Crohn’s disease attacks the wall of the intestines and makes it very difficult to digest food properly. Peter has the option to undergo surgery (ileostomy), which will have a major impact on his life. The task of the role-play is to discuss the pros and cons with friends and family and decide whether he should have the operation. The other characters are the following: Janet (mom) who wants Peter to have the operation, Matthew (older brother) who is against the operation, Arnold (Dad) who is not able to face the situation and David (the best friend) who mediates the discussion.

  2. The bully, Mayid, is picking on a new schoolmate, Lisa. Elise and Dave (Lisa’s friends), and Mrs Parton (the school teacher) are trying to stop the bullying.

References

  1. Zhang L. Exploitation of contextual affect-sensing and dynamic relationship interpretation. ACM Comput Entertaint. 2010; 8(3) (article no.: 18).

  2. Ortony A, Clore GL, Collins A. The cognitive structure of emotions. Cambridge: Cambridge University Press; 1998.

    Google Scholar 

  3. Ekman P. An argument for basic emotions. Cogn Emot. 1992;6:169–200.

    Article  Google Scholar 

  4. Jess, the rule engine for the Java platform. http://www.jessrules.com/. Accessed in Feb 2012.

  5. Briscoe E, Carroll J. Robust accurate statistical annotation of general text. In: Proceedings of the 3rd international conference on language resources and evaluation, Las Palmas, Gran Canaria; 2002. p. 1499–1504.

  6. Zhang L, Barnden JA, Hendley RJ, Wallington AM. Exploitation in affect detection in improvisational E-drama. In: Proceedings of the 6th international conference on intelligent virtual agents. California, USA. Lecture notes in computer science. 2006;4133:68–79. Springer.

  7. Zhang L, Barnden JA, Hendley RJ, Lee MG, Wallington AM, Wen Z. Affect detection and metaphor in E-drama. Cont Eng Educ Life-Long Learn. 2008;18(2):234–52.

    Article  Google Scholar 

  8. Zhang L, Gillies M, Dhaliwal K, Gower A, Robertson D, Crabtree B. E-drama: facilitating online role-play using an AI actor and emotionally expressive characters. Int J Artif Intellig Educ. 2009;19(1):5–38.

    CAS  Google Scholar 

  9. Kappas A. Smile when you read this, whether you like it or not: Conceptual challenges to affect detection. IEEE Transact Affect Comput. 2010;1(1):38–41. doi:10.1109/T-AFFC.2010.6.

  10. Hareli S, Rafaeli A. Emotion cycles: on the social influence of emotion in organizations. Res Organ Behav. 2008;28:35–59.

    Article  Google Scholar 

  11. Picard RW. Affective computing. Cambridge MA: The MIT Press; 2000.

    Google Scholar 

  12. Prendinger H, Ishizuka M. Simulating affective communication with animated agents. In: Proceedings of Eighth IFIP TC.13 conference on human–computer interaction. Tokyo, Japan. 2001, p. 182–189.

  13. Mehdi EJ, Nico P, Julie D, Bernard P. Modeling character emotion in an interactive virtual environment. In: Proceedings of AISB 2004 symposium: motion, emotion and cognition. Leeds, UK; 2004.

  14. Gratch J, Marsella S. A domain-independent framework for modeling emotion. J Cogn Syst Res. 2004;5:269–306.

    Article  Google Scholar 

  15. Aylett A, Louchart S, Dias J, Paiva A, Vala M, Woods S et al. Unscripted narrative for affectively driven characters. IEEE Comput Graphics Applicat 2006;26(3):42–52.

    Google Scholar 

  16. Endrass B, Rehm M, André E. Planning small talk behavior with cultural influences for multiagent systems. Comput Speech Lang. 2011;25(2):158–74.

    Article  Google Scholar 

  17. Liu H, Singh P, ConceptNet. A practical commonsense reasoning toolkit. BT Technology Journal, Volume 22, Kluwer Academic Publishers; 2004.

  18. Neviarouskaya A, Prendinger H, Ishizuka M. Recognition of affect, judgment, and appreciation in text. In: Proceedings of the 23rd international conference on computational linguistics, Beijing, China; 2010:806–814.

  19. Mateas M. Ph.D. thesis. Interactive drama, art and artificial intelligence. School of Computer Science, Carnegie Mellon University; 2002.

  20. Zhe X, Boucouvalas AC. Text-to-emotion engine for real time internet communication. In: Proceedings of international symposium on communication systems, Networks and DSPs, Staffordshire University, UK; 2002:164–168.

  21. Ptaszynski M, Dybala P, Shi W, Rzepka, R, Araki K. Towards context aware emotional intelligence in machines: computing contextual appropriateness of affective states. In: Proceeding of IJCAI; 2009.

  22. Craggs R, Wood M. A two dimensional annotation scheme for emotion in dialogue. In: Proceedings of AAAI spring symposium: exploring attitude and affect in text; 2004.

  23. Werry C. Linguistic and interactional features of internet relay chat. In computer-mediated communication: linguistic: social and cross-cultural perspectives. Pragmatics and beyond new series 39. Amsterdam: John Benjamins; 1996:47–64.

  24. Bouckaert RR, Frank E, Hall M, Holmes G, Pfahringer B, Reutemann P, Witten IH. WEKA-experiences with a java open-source project. J Mach Leran Res. 2010;11:2533–41.

    Google Scholar 

  25. Landauer TK, Dumais S. Latent semantic analysis. Scholarpedia. 2008;3(11):4356.

    Article  Google Scholar 

  26. Widdows D, Cohen T. The semantic vectors package: new algorithms and public tools for distributional semantics. The fourth ieee international conference on semantic computing (IEEE ICSC2010); 2010.

  27. Zhang L, Barnden JA. Affect and Metaphor Sensing in Virtual Drama. Int J Comput Games Technol. 2010; 2010 (article ID 512563).

  28. ATT-Meta project databank. http://www.cs.bham.ac.uk/~jab/ATT-Meta/Databank/. Assessed March 2012.

  29. Rabiner LR. A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE. 1989;77(2):257–86.

    Article  Google Scholar 

  30. Wang Z, Lee J, Marsella S. Towards More comprehensive listening behavior: beyond the bobble head. In: Proceedings of the 10th international conference on intelligent virtual agents. Iceland; 2011.

Download references

Acknowledgments

We thank Dr. Meurig Beynon for his help with proofreading and detailed feedback on the technical aspects and manner of writing. We also thank Dr. Zach Pardos and Dr. Tristan Nixon, the organizers of a tutorial, Parameter Fitting for Learner Models, associated with 2012 International Conference on Intelligent Tutoring Systems.

Author information

Authors and Affiliations

  1. School of Computing, Engineering and Information Sciences, Northumbria University, Newcastle, NE2 1XE, UK

    Li Zhang

  2. School of Computer Science, University of Birmingham, Birmingham, B15 2TT, UK

    John Barnden

Authors
  1. Li Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Barnden
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Zhang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, L., Barnden, J. Affect Sensing Using Linguistic, Semantic and Cognitive Cues in Multi-threaded Improvisational Dialogue. Cogn Comput 4, 436–459 (2012). https://doi.org/10.1007/s12559-012-9170-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-012-9170-3

Keywords

Search

Navigation