Context-Aware Multimodal Sharing of Emotions

  • Maurizio Caon
  • Leonardo Angelini
  • Yong Yue
  • Omar Abou Khaled
  • Elena Mugellini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8008)


Computer mediated interaction often lacks of expressivity, in particular for emotion communication. Therefore, we present a concept for context-aware multimodal sharing of emotions for human-to-computer-to-human interaction in social networks. The multimodal inputs and outputs of this system are distributed in a smart environment in order to grant a more immersive and natural interaction experience. The context information is used to improve the opportuneness and the quality of feedback. We implemented an evaluation scenario and we conducted an observation study during some events with the participants. We reported our considerations at the end of this paper.


affective computing multimodal interaction computer mediated communication social sharing of emotions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Clubb, O.L.: Human-to-Computer-to-Human Interactions (HCHI) of the communications revolution. Interactions 14(2), 35 (2007)CrossRefGoogle Scholar
  2. 2.
    Kivran-Swaine, F., Naaman, M.: Network properties and social sharing of emotions in social awareness streams. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, p. 379 (July 2011)Google Scholar
  3. 3.
    Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: A survey of principles, models and frameworks. In: Lalanne, D., Kohlas, J. (eds.) Human Machine Interaction. LNCS, vol. 5440, pp. 3–26. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  4. 4.
    Mugellini, E., Abou Khaled, O., Pierroz, S., Carrino, S., Chabbi Drissi, H.: Generic Framework for Transforming Everyday Objects into Interactive Surfaces. In: Jacko, J.A. (ed.) HCI International 2009, Part III. LNCS, vol. 5612, pp. 473–482. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  5. 5.
    Emotiv EPOC neuroheadset, (last visited: March 2013)
  6. 6.
    Perroud, D., Angelini, L., Khaled, O.A., Mugellini, E.: Context-Based Generation of Multimodal Feedbacks for Natural Interaction in Smart Environments. In: Proceedings of the Second International Conference on Ambient Computing, Applications, Services and Technologies, pp. 19–25 (2012)Google Scholar
  7. 7.
    Aphrodite project, (last visited: March 2013)
  8. 8.
    Phidgets sensors, (last visited: March 2013)
  9. 9.
    Hwang, H.C., Matsumoto, D.: Nonverbal Behaviors and Cross-Cultural Communication in the New Era. Language and Intercultural Communication in the New Era, 116 (2013)Google Scholar
  10. 10.
    Rimé, B., Finkenauer, C., Luminet, O., Zech, E., Philippot, P.: Social Sharing of Emotion: New Evidence and New Questions. European Review of Social Psychology 9, 145–189 (1998)CrossRefGoogle Scholar
  11. 11.
    Hudlicka, E.: To feel or not to feel: The role of affect in human–computer interaction. International Journal of Human-Computer Studies 59, 1–32 (2003)CrossRefGoogle Scholar
  12. 12.
    Hoffman, D.L., Novak, T.P., Venkatesh, A.: Has the Internet become indispensable? Communications of the ACM 47, 37–42 (2004)CrossRefGoogle Scholar
  13. 13.
    Picard, R.W.: Does HAL cry digital tears? Emotions and computers. In: Hal’s Legacy: 2001’s Computer as Dream and Reality, pp. 279–303 (1997)Google Scholar
  14. 14.
    Brave, S., Nass, C.: Emotion in human–computer interaction. In: The Human–Computer Interaction Handbook, pp. 81–93 (2003)Google Scholar
  15. 15.
    Picard, R.W.: Affective Computing. The MIT Press (1997)Google Scholar
  16. 16.
    Kleinsmith, A., Bianchi-Berthouze, N.: Affective Body Expression Perception and Recognition: A Survey. IEEE Transactions on Affective Computing, 1 (2012)Google Scholar
  17. 17.
    Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3D facial expression recognition: A comprehensive survey. Image and Vision Computing 30, 683–697 (2012)CrossRefGoogle Scholar
  18. 18.
    Anagnostopoulos, C.N., Iliou, T., Giannoukos, I.: Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011. Artificial Intelligence Review, 1–23 (2012)Google Scholar
  19. 19.
    Pantic, M., Sebe, N., Cohn, J.F., Huang, T.: Affective multimodal human-computer interaction. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, p. 669 (2005)Google Scholar
  20. 20.
    Canento, F., Fred, A., Silva, H., Gamboa, H., Lourenco, A.: Multimodal biosignal sensor data handling for emotion recognition. In: Proceedings of 2011 IEEE SENSORS, pp. 647–650 (2011)Google Scholar
  21. 21.
    Schaaff, K., Schultz, T.: Towards emotion recognition from electroencephalographic signals. In: Proceedings of the 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pp. 1–6 (2009)Google Scholar
  22. 22.
    Benini, S., Canini, L., Leonardi, R.: A Connotative Space for Supporting Movie Affective Recommendation. IEEE Transactions on Multimedia 13, 1356–1370 (2011)CrossRefGoogle Scholar
  23. 23.
    Dornbush, S., Fisher, K., McKay, K., Prikhodko, A., Segall, Z.: XPOD - A Human Activity and Emotion Aware Mobile Music Player. In: Proceedings of the 2nd Asia Pacific Conference on Mobile Technology, Applications and Systems, pp. 1–6 (2005)Google Scholar
  24. 24.
    Tsai, T.W., Lo, H.Y., Chen, K.S.: An affective computing approach to develop the game-based adaptive learning material for the elementary students. In: Proceedings of the 2012 Joint International Conference on Human-Centered Computer Environments, p. 8 (2012)Google Scholar
  25. 25.
    Mihelj, M., Novak, D., Munih, M.: Emotion-aware system for upper extremity rehabilitation. In: Proceedings of Virtual Rehabilitation International Conference, pp. 160–165 (2009)Google Scholar
  26. 26.
    Paiva, A.: Affective interactions: towards a new generation of computer interfaces. In: Paiva, A.M. (ed.) Affective Interactions. LNCS (LNAI), vol. 1814, pp. 1–8. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  27. 27.
    Picard, R.W., Cosier, G.: Affective intelligence — the missing link? BT Technology Journal (1997)Google Scholar
  28. 28.
    Tsujita, H., Rekimoto, J.: Smiling makes us happier. In: Proceedings of the 13th International Conference on Ubiquitous Computing (2011)Google Scholar
  29. 29.
    McDuff, D., Kaliouby, R.E., Picard, R.W.: Crowdsourcing Facial Responses to Online Videos. IEEE Transactions on Affective Computing 3, 456–468 (2012)CrossRefGoogle Scholar
  30. 30.
    Teh, J.K.S., Cheok, A.D., Peiris, R.L., Choi, Y., Thuong, V., Lai, S.: Huggy Pajama. In: Proceedings of the 7th International Conference on Interaction Design and Children, p. 250 (2008)Google Scholar
  31. 31.
    Wang, R., Quek, F., Tatar, D., Teh, K.S., Cheok, A.: Keep in touch. In: Proceedings of the 2012 ACM Annual Conference on Human Factors in Computing Systems, p. 139 (2012)Google Scholar
  32. 32.
    Lisetti, C., Nasoz, F., LeRouge, C., Ozyer, O., Alvarez, K.: Developing multimodal intelligent affective interfaces for tele-home health care. International Journal of Human-Computer Studies 59, 245–255 (2003)CrossRefGoogle Scholar
  33. 33.
    Willis, M.J., Jones, C.M.: Emotishare: emotion sharing on mobile devices. In: Proceedings of the 26th Annual BCS Interaction Specialist Group Conference on People and Computers (2012)Google Scholar
  34. 34.
    Picard, R.W.: Affective Computing: From Laughter to IEEE. IEEE Transactions on Affective Computing 1, 11–17 (2010)CrossRefGoogle Scholar
  35. 35.
    Moser, E., Derntl, B., Robinson, S., Fink, B., Gur, R.C., Grammer, K.: Amygdala activation at 3T in response to human and avatar facial expressions of emotions. Journal of Neuroscience Methods 161, 126–133 (2007)CrossRefGoogle Scholar
  36. 36.
    Plutchik, R.: The Nature of Emotions. American Scientist 89(4), 344–350 (2001)Google Scholar
  37. 37.
    Affectiva Q-Sensor, (last visited: March 2013)
  38. 38.
    Interaxon headset, (last visited: March 2013)
  39. 39.
    Socialize event at Fribourg, (last visited: March 2013)

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Maurizio Caon
    • 1
    • 2
  • Leonardo Angelini
    • 1
  • Yong Yue
    • 2
  • Omar Abou Khaled
    • 1
  • Elena Mugellini
    • 1
  1. 1.University of Applied Sciences of Western SwitzerlandFribourgSwitzerland
  2. 2.University of BedfordshireLutonUK

Personalised recommendations