Skip to main content
SpringerLink
Log in
Book cover

Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces. Theoretical and Practical Issues pp 81–95Cite as

Sentic Avatar: Multimodal Affective Conversational Agent with Common Sense

  • Chapter

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 6456))

Abstract

The capability of perceiving and expressing emotions through different modalities is a key issue for the enhancement of human-computer interaction. In this paper we present a novel architecture for the development of intelligent multimodal affective interfaces. It is based on the integration of Sentic Computing, a new opinion mining and sentiment analysis paradigm based on AI and Semantic Web techniques, with a facial emotional classifier and Maxine, a powerful multimodal animation engine for managing virtual agents and 3D scenarios. One of the main distinguishing features of the system is that it does not simply perform emotional classification in terms of a set of discrete emotional labels but it operates in a continuous 2D emotional space, enabling the integration of the different affective extraction modules in a simple and scalable way.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Vesterinen, E.: Affective Computing. In: Digital Media Research Seminar, Helsinki (2001)

    Google Scholar 

  2. Pantic, M.: Affective Computing. Encyclopedia of Multimedia Technology and Networking. Hershy 1, 8–14, Idea Group Reference (2005)

    Article  Google Scholar 

  3. Gunes, M., Gunes, H., Piccardi, M., Pantic, M.: From the Lab to the Real World: Affect Recognition Using Multiple Cues and Modalities. In: Affective Computing: Focus on Emotion Expression, Synthesis and Recognition, pp. 185–218 (2008)

    Google Scholar 

  4. Riseberg, J., Klein, J., Fernandez, R., Picard, R.: Frustrating the User on Purpose: Using Biosignals in a Pilot Study to Detect the User’s Emotional State. In: CHI, Los Angeles (1998)

    Google Scholar 

  5. Ambady, N., Rosenthal, R.: Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences: a Meta-Analysis. Psychological Bulletin 11(2), 256–274 (1992)

    Article  Google Scholar 

  6. Camurri, A., Mazzarino, B., Volpe, G.: Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library. In: Gesture Workshop, Genova (2003)

    Google Scholar 

  7. Gunes, H., Piccardi. M.: Bi-Modal Emotion Recognition from Expressive Face and Body Gestures. Network and Computer Applications 30(4), 1334–1345 (2007)

    Article  Google Scholar 

  8. Karpouzis, K., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A., Malatesta, L., Kollias, S.: Modeling Naturalistic Affective States Via Facial, Vocal and Bodily Expressions Recognition. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds.) ICMI/IJCAI Workshops 2007. LNCS (LNAI), vol. 4451, pp. 92–116. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  9. Pun, T., Alecu, T., Chanel, G., Kronegg, J., Voloshynovskiy, S.: Brain-Computer Interaction Research at the Computer Vision and Multimedia Laboratory. IEEE Trans. on Neural Systems and Rehabilitation Engineering 14(2), 210–213 (2006)

    Article  Google Scholar 

  10. Burleson, W., Picard, R., Perlin, K., Lippincott, J.: A Platform for Affective Agent Research. In: International Conference on Autonomous Agents and Multiagent Systems, New York (2004)

    Google Scholar 

  11. Petridis, S., Pantic, M.: Audiovisual Discrimination between Laughter and Speech. In: ICASSP, Las Vegas (2008)

    Google Scholar 

  12. Valstar, M., Gunes, H., Pantic, M.: How to Distinguish Posed from Spontaneous Smiles Using Geometric Features. In: ICMI, Nagoya (2007)

    Google Scholar 

  13. Truong, K., Van Leeuwen, D.: Automatic Discrimination Between Laughter and Speech. Speech Communication 49, 144–158 (2007)

    Article  Google Scholar 

  14. Matos, S., Birring, S., Pavord, I., Evans, D.: Detection of Cough Signals in Continuous Audio Recordings Using HMM. IEEE Trans. on Biomedical Engineering 53(6), 1078–1083 (2006)

    Article  Google Scholar 

  15. Pal, P., Iyer, A., Yantorno, R.: Emotion Detection from Infant Facial Expressions and Cries. In: Intl Conf. Acoustics, Speech and Signal Processing (2006)

    Google Scholar 

  16. Jong-Tae, J., Sang-Wook, S., Kwang-Eun, K., Kwee-Bo, S.: Emotion Recognition Method Based on Multimodal Sensor Fusion Algorithm. In: ISIS, Sokcho-City (2007)

    Google Scholar 

  17. Shan, C., Gong, S., McOwan, P.: Beyond Facial Expressions: Learning Human Emotion from Body Gestures. In: BMVC, Warwick (2007)

    Google Scholar 

  18. Baldassarri, S., Cerezo, E., Seron, F.: Maxine: a Platform for Embodied Animated Agents. Computers and Graphics 32(4), 430–437 (2008)

    Article  Google Scholar 

  19. Cambria, E., Hussain, A., Havasi, C., Eckl, C.: Sentic Computing: Exploitation of Common Sense for the Development of Emotion-Sensitive Systems. In: Esposito, A., Campbell, N., Vogel, C., Hussain, A., Nijholt, A. (eds.) COST 2102. LNCS, vol. 5967, pp. 153–161. Springer, Heidelberg (2010)

    Google Scholar 

  20. Cambria, E., Hussain, A., Havasi, C., Eckl, C.: Common Sense Computing: From the Society of Mind to Digital Intuition and Beyond. In: Fierrez, J., Ortega-Garcia, J., Esposito, A., Drygajlo, A., Faundez-Zanuy, M. (eds.) BioID MultiComm 2009. LNCS, vol. 5707, pp. 252–259. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  21. Cambria, E., Grassi, M., Hussain, A., Havasi, C.: Sentic Computing for Social Media Marketing. In: Multimedia Tools and Applications. Springer, Heidelberg (to appear, 2010)

    Google Scholar 

  22. Elliott, C.: The Affective Reasoner: A Process Model of Emotions in a Multi-Agent System. The Institute for the Learning Sciences, Technical Report No. 32 (1992)

    Google Scholar 

  23. Ortony, A., Clore, G., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, New York (1988)

    Book  Google Scholar 

  24. Wiebe, J., Wilson, T., Claire, C.: Annotating Expressions of Opinions and Emotions in Language. Language Resources and Evaluation 39(2), 165–210 (2005)

    Article  Google Scholar 

  25. Wilson, T., Wiebe, J., Hoffmann, P.: Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. In: HLT-EMNLP, Vancouver (2005)

    Google Scholar 

  26. Somasundaran, S., Wiebe, J., Ruppenhofer, J.: Discourse Level Opinion Interpretation. In: COLING, Manchester (2008)

    Google Scholar 

  27. Hu, M., Liu, B.: Mining Opinion Features in Customer Reviews. In: AAAI, San Jose (2004)

    Google Scholar 

  28. Pang, B., Lee, L.: Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales. In: ACL, Ann Arbor (2005)

    Google Scholar 

  29. Abbasi, A., Chen, H., Salem, A.: Sentiment Analysis in Multiple Languages: Feature Selection for Opinion Classification in Web Forums. ACM Transactions on Information Systems 26(3), 1–34 (2008)

    Article  Google Scholar 

  30. Cambria, E., Hussain, A., Havasi, C., Eckl, C.: AffectiveSpace: Blending Common Sense and Affective Knowledge to Perform Emotive Reasoning. In: WOMSA at CAEPIA, Seville (2009)

    Google Scholar 

  31. Havasi, C., Speer, R., Alonso, J.: ConceptNet 3: a Flexible, Multilingual Semantic Network for Common Sense Knowledge. In: RANLP, Borovets (2007)

    Google Scholar 

  32. Strapparava, C., Valitutti, A.: WordNet-Affect: an Affective Extension of WordNet. In: LREC, Lisbon (2004)

    Google Scholar 

  33. Havasi, C., Speer, R., Pustejovsky, J., Lieberman, H.: Digital Intuition: Applying Common Sense Using Dimensionality Reduction. IEEE Intelligent Systems 24(4), 24–35 (2009)

    Article  Google Scholar 

  34. Wall, M., Rechtsteiner, A., Rocha, L.: Singular Value Decomposition and Principal Component Analysis. In: Berrar, D., et al. (eds.) A Practical Approach to Microarray Data Analysis, pp. 91–109. Kluwer, Norwell (2003)

    Chapter  Google Scholar 

  35. Eckart, C., Young, G.: The Approximation of One Matrix by Another of Lower Rank. Psychometrika 1(3), 211–218 (1936)

    Article  MATH  Google Scholar 

  36. Plutchik, R.: The Nature of Emotions. American Scientist 89(4), 344–350 (2001)

    Article  Google Scholar 

  37. Minsky, M.: The Emotion Machine. Simon and Schuster, New York (2006)

    Google Scholar 

  38. Loquendo Audio Speech Recognition, http://www.loquendo.com

  39. Ekman, P., Dalgleish, T., Power, M.: Handbook of Cognition and Emotion. Wiley, Chihester (1999)

    Google Scholar 

  40. Cerezo, E., Hupont, I., Manresa, C., Varona, J., Baldassarri, S., Perales, F., Seron, F.: Real-Time Facial Expression Recognition for Natural Interaction. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds.) IbPRIA 2007. LNCS, vol. 4478, pp. 40–47. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  41. Witten, I., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco (2005)

    MATH  Google Scholar 

  42. Wallhoff, F.: Facial Expressions and Emotion Database. Technische Universitat Munchen (2006)

    Google Scholar 

  43. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-Based Database for Facial Expression Analysis. In: ICME, Singapore (2005)

    Google Scholar 

  44. Siegel, S., Castellan, N.: Nonparametric Statistics for the Social Siences. McGraw-Hill, New York (1988)

    Google Scholar 

  45. Whissell, C.: The Dictionary of Affect in Language. Emotion: Theory, Research and Experience, The Measurement of Emotions 4, 113–131 (1989)

    Google Scholar 

  46. Plutchik, R.: Emotion: a Psychoevolutionary Synthesis. Harper and Row, New York (1980)

    Google Scholar 

  47. Kumar, P., Yildirim, E.: Minimum-Volume Enclosing Ellipsoids and Core Sets. Journal of Optimization Theory and Applications 126, 1–21 (2005)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Stirling, Stirling, UK

    Erik Cambria & Amir Hussain

  2. Aragon Institute of Technology, Zaragoza, Spain

    Isabelle Hupont

  3. University of Zaragoza, Zaragoza, Spain

    Eva Cerezo & Sandra Baldassarri

Authors
  1. Erik Cambria
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Isabelle Hupont
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amir Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eva Cerezo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sandra Baldassarri
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute for Advanced Scientific Studies, Second University of Naples, and IIASS, Via Pellegrino 19, 84019, Vietri sul Mare (SA), Italy

    Anna Esposito

  2. Istituto Nazionale di Geofisica e Vulcanologia, Osservatorio Vesuviano, Via Diocleziano 328, 80124, Napoli, Italy

    Antonietta M. Esposito

  3. Dipartemento di Ingegneria dell’ Informazione, Seconda Università di Napoli, Via Roma 29, 81031, Aversa (CE), Italy

    Raffaele Martone

  4. Department of Humanities and Social Sciences, Anatolia College/ACT, Kennedy Street, 55510, Pylaia, Greece

    Vincent C. Müller

  5. Departmnet of Physics "E.R. Caoamoeööp", University of Salerno and IIASS, International Institute for Advanced Scientific Studies, 84081, Baronissi (SA), Italy

    Gaetano Scarpetta

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Cambria, E., Hupont, I., Hussain, A., Cerezo, E., Baldassarri, S. (2011). Sentic Avatar: Multimodal Affective Conversational Agent with Common Sense. In: Esposito, A., Esposito, A.M., Martone, R., Müller, V.C., Scarpetta, G. (eds) Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces. Theoretical and Practical Issues. Lecture Notes in Computer Science, vol 6456. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-18184-9_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-18184-9_8

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-18183-2

  • Online ISBN: 978-3-642-18184-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation