Recognizing the User Social Attitude in Multimodal Interaction in Smart Environments

  • Berardina De Carolis
  • Stefano Ferilli
  • Nicole Novielli
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7683)


Ambient Intelligence aims at promoting an effective, natural and personalized interaction with the environment services. In order to provide the most appropriate answer to the user requests, an Ambient Intelligence system should model the user by considering not only the cognitive ingredients of his mental state, but also extra-rational factors such as affect, engagement, attitude, and so on. This paper describes a study aimed at building a multimodal framework for recognizing the social response of users during interaction with embodied agents in the context of ambient intelligence. In particular, we describe how we extended a model for recognizing the social attitude in text-based dialogs by adding two additional knowledge sources: speech and gestures. Results of the study show that these additional knowledge sources may help in improving the recognition of the users’ attitude during interaction.


Social Attitude Gesture Recognition Dynamic Time Warping Social Robot Dynamic Belief Network 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Riva, G., Vatalaro, F., Davide, F., Alcaniz, M. (eds.): Ambient Intelligence: the Evolution of Technology, Communication and Cognition Towards the Future of Human-Computer Interaction Emerging Communication (2005)Google Scholar
  2. 2.
    López-Cózar, R., Callejas, Z.: Multimodal Dialogue for Ambient Intelligence and Smart Environments. In: Nakashima, H., Aghajan, H., Augusto, J.C. (eds.) Intelligence, p. 93808. Springer US (2010)Google Scholar
  3. 3.
    Nijholt, A., de Ruyter, B., Heylen, D., Privender, S.: Social Interfaces for Ambient Intelligence Environments. In: Aarts, E., Encarnaçao, J. (eds.) True Visions: The Emergence of Ambient Intelligence, ch. 14, pp. 275A–289A. Springer, New York (2006)Google Scholar
  4. 4.
    Ortiz, A., del Puy Carretero, M., Oyarzun, D., Yanguas, J.J., Buiza, C., Gonzalez, M.F., Etxeberria, I.: Elderly Users in Ambient Intelligence: Does an Avatar Improve the Interaction? In: Stephanidis, C., Pieper, M. (eds.) ERCIM UI4ALL Ws 2006. LNCS, vol. 4397, pp. 99–114. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Cesta, A., Cortellessa, G., Giuliani, M.V., Iocchi, L., Leone, G.R., Nardi, D., Pecora, F., Rasconi, R., Scopelliti, M., Tiberio, L.: The RoboCare Assistive Home Robot: Environment, Features and Evaluation, The RoboCare Technical Report, RC-TR-0906-6 (2006)Google Scholar
  6. 6.
    van Breemen, J.N.: iCat: a generic platform for studying personal robot applications. Paper Presented at the IEEE SMC, Den Haag (2004)Google Scholar
  7. 7.
    Reeves, B., Nass, C.: The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, Cambridge (1996)Google Scholar
  8. 8.
    Bickmore, T., Cassell, J.: Social Dialogue with Embodied Conversational Agents. In: van Kuppevelt, J., Dybkjaer, L., Bernsen, N. (eds.) Advances in Natural, Multimodal Dialogue Systems, pp. 1–32. Kluwer Academic, New York (2005)Google Scholar
  9. 9.
    Cassell, J., Prevost, S., Sullivan, J., Churchill, E.: Embodied Conversational Agents. MIT Press, Cambridge (2000)Google Scholar
  10. 10.
    Paiva, A. (ed.): Empathic Agents. Workshop in conjunction with AAMAS 2004 (2005)Google Scholar
  11. 11.
    Jensen, F.V.: Bayesian Networks and Decision Graphs. Springer (2001)Google Scholar
  12. 12.
    Whittaker, S., Walker, M., Moore, J.: Fish or Fowl: A Wizard of Oz evaluation of dialogue strategies in the restaurant domain. In: Language Resources and Evaluation Conference (2002)Google Scholar
  13. 13.
    Oviatt, S., Adams, B.: Designing and Evaluating Conversational Interfaces With Animated Characters. In: Cassell, J., Sullivan, J., Prevost, S., Churchill, E. (eds.) Embodied Conversational Agents. The MIT Press (2000)Google Scholar
  14. 14.
    Novielli, N., de Rosis, F., Mazzotta, I.: User Attitude Towards an Embodied Conversational Agent: Effects of the Interaction Mode. Journal of Pragmatics 42(9), 2385–2397 (2010) ISSN: 0378-2166, doi:10.1016/j.pragma.2009.12.016CrossRefGoogle Scholar
  15. 15.
    Andersen, P.A., Guerrero, L.K.: Handbook of Communication and Emotions. Research, theory, applications and contexts. Academic Press (1998)Google Scholar
  16. 16.
    Polhemus, L., Shih, L.-F., Swan, K.: Virtual interactivity: the representation of social presence in an on line discussion. In: Annual Meeting of the American Educational Research Association (2001)Google Scholar
  17. 17.
    Swan, K.: Immediacy, social presence and asynchronous discussion. In: Bourne, J., Moore, J.C. (eds.) Elements of Quality Online Education, vol. 3. Sloan Center For Online Education, Nedham (2002)Google Scholar
  18. 18.
  19. 19.
  20. 20.
    Markopoulos, P.: Designing ubiquitous computer human interaction: the case of the connected family. In: Isomaki, H., Pirhonen, A., Roast, C., Saariluoma, P. (eds.) Future Interaction Design. Springer (2004)Google Scholar
  21. 21.
    Norman, D.: The Invisible Computer. MIT Press, Cambridge (1998)Google Scholar
  22. 22.
    Pentland, A.: Perceptual intelligence. Communications of the ACM 43(3), 35–44 (2000)CrossRefGoogle Scholar
  23. 23.
    Reilly, W.S.N.: Believable Social and Emotional Agents. PhD thesis (1996)Google Scholar
  24. 24.
    Brahnam, S., De Angeli, A.: Special issue on the abuse and misuse of social agents. Interacting with Computers 20(3), 287–291 (2008)CrossRefGoogle Scholar
  25. 25.
    Bickmore, T.: Relational Agents: Effecting Change through Human-Computer Relationships. PhD Thesis, Media Arts & Sciences, Massachusetts Institute of Technology (2003)Google Scholar
  26. 26.
    Hoorn, J.F., Konijn, E.A.: Perceiving and Experiencing Fictitional Characters: An integrative account. Japanese Psychological Research 45, 4 (2003)CrossRefGoogle Scholar
  27. 27.
    Cassell, J., Bickmore, T.: Negotiated collusion: modelling social language and its relationship effects in intelligent agents. User Modelling and User-Adapted Interaction 13, 1–2 (2003)Google Scholar
  28. 28.
    Bailenson, J.N., Aharoni, E., Beall, A.C., Guadagno, R.E., Dimov, A., Blascovich, J.: Comparing behavioral and self-report measures of embodied agents’ social presence in immersive virtual environments. In: Proceedings of 7th Annual International Workshop on PRESENCE, pp. 216–223 (2004)Google Scholar
  29. 29.
    Bailenson, J.N., Swinth, K.R., Hoyt, C.L., Persky, S., Dimov, A., Blascovich, J.: The independent and interactive effects of embodied agents appearance and behavior on self-report, cognitive and behavioral markers of copresence in Immersive Virtual Environments. Presence 14(4), 379–393 (2005)CrossRefGoogle Scholar
  30. 30.
    Cramer, H., Kemper, N., Amin, A., Wielinga, B., Evers, V.: Give me a hug: the effects of touch and autonomy on people’s responses to embodied social agents. Comp. Anim. Virtual Worlds 20, 437–445 (2009), doi:10.1002/cav.317CrossRefGoogle Scholar
  31. 31.
    Mozziconacci, S., Hermes, D.J.: Role of intonation patterns in conveying emotion in speech. In: Proceedings, International Conference of Phonetic Sciences, San Francisco (1999)Google Scholar
  32. 32.
    Bosma, W.E., André, E.: Exploiting Emotions to Disambiguate Dialogue Acts. In: Nunes, N.J., Rich, C. (eds.) Proc. 2004 Conference on Intelligent User Interfaces, Funchal, Portugal, January 13, pp. 85–92 (2004)Google Scholar
  33. 33.
    Litman, D., Forbes, K., Silliman, S.: Towards emotion prediction in spoken tutoring dialogues. In: Proceedings of HLT/NAACL (2003)Google Scholar
  34. 34.
    De Carolis, B., Cozzolongo, G.: Interpretation of User’s Feedback in Human-Robot Interaction. Journal of Physical Agents 3(2), 47–58 (2009) ISSN: 1888-0258Google Scholar
  35. 35.
    Sundberg, J., Patel, S., Björkner, E., Scherer, K.R.: Interdependencies among voice source parameters in emotional speech. IEEE Transactions on Affective Computing 2(3) (2011)Google Scholar
  36. 36.
    Vogt, T., André, E., Bee, N.: EmoVoice — A Framework for Online Recognition of Emotions from Voice. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS (LNAI), vol. 5078, pp. 188–199. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  37. 37.
    Allan, Pease, B.: The Definitive Book of Body Language. Bantam Books (2006)Google Scholar
  38. 38.
    Feldman, R.S.: Applications of Nonverbal Behavioral Theories and Research. Lawrence Erlbaum Associates, Inc., New Jersey (1992)Google Scholar
  39. 39.
    Knapp, M., Hall, J.: Nonverbal communication in human interaction. Holt, Rinehart & Winsten, Inc., Orlando (1992)Google Scholar
  40. 40.
    Malandro, L.A.: Non-verbal Communication, 2nd edn. Random House, New York (1983,1989)Google Scholar
  41. 41.
    Efron, D.: Gesture, Race and Culture. Mouton and Co. (1972)Google Scholar
  42. 42.
    Rehm, M., Nakano, Y., André, E., Nishida, T.: Culture-Specific First Meeting Encounters between Virtual Agents. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 223–236. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  43. 43.
    Prendinger, H., Mori, J., Ishizuka, M.: Recognizing, Modeling, and Responding to Users’ Affective States. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 60–69. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  44. 44.
    Conati, C.: Probabilistic assessment of user’s emotions in educational games. Applied Artificial Intelligence 16, 555–575 (2002)CrossRefGoogle Scholar
  45. 45.
    Sabourin, J., Mott, B., Lester, J.: Computational Models of Affect and Empathy for Pedagogical Virtual Agents. Standards in Emotion Modeling, Lorentz Center International Center for Workshops in the Sciences (2011)Google Scholar
  46. 46.
    Caridakis, G., Karpouzis, K., Wallace, M., Kessous, L., Amir, N.: Multimodal user’s affective state analysis in naturalistic interaction. Journal on Multimodal User Interfaces 3(1-2), 49–66 (2009)CrossRefGoogle Scholar
  47. 47.
    Li, X.: Integrating User Affective State Assessment in Enhancing HCI: Review and Proposition. The Open Cybernetics and Systemics Journal 2, 192–205 (2008)CrossRefGoogle Scholar
  48. 48.
    Boersma, P., Weenink, D.: Praat: doing phonetics by computer (version 4.5.15) [computer program] (2007), (retrieved February 24, 2007)
  49. 49.
    Martin, B.: Instance-Based learning: Nearest Neighbor With Generalization, Master Thesis, University of Waikato, Hamilton, New Zealand (1995)Google Scholar
  50. 50.
    Kistler, F., Endrass, B., Damian, I., Dang, C.T., André, E.: Natural interaction with culturally adaptive virtual characters. Journal on Multimodal User Interfaces Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Berardina De Carolis
    • 1
  • Stefano Ferilli
    • 1
  • Nicole Novielli
    • 1
  1. 1.Dipartimento di InformaticaUniversity of BariItaly

Personalised recommendations