Soft Computing

, Volume 21, Issue 18, pp 5309–5323 | Cite as

Assessing users’ emotion at interaction time: a multimodal approach with multiple sensors

  • Vinícius P. Gonçalves
  • Gabriel T. Giancristofaro
  • Geraldo P. R. Filho
  • Thienne Johnson
  • Valéria Carvalho
  • Gustavo Pessin
  • Vânia P. de Almeida Neris
  • Jó Ueyama
Methodologies and Application


Users’ emotional states influence decision making and are essential for the knowledge and explanation of users’ behavior with computer applications. However, collecting emotional states during the interaction time with users is a onerous task because it requires very careful handling of the empirical observation, leading researchers to carry out assessments of emotional responses only at the end of the interaction. This paper reports our research in assessing users’ behavior at interaction time and also describes the results of a case study which analyzed users’ emotional responses while interacting with a game. We argue that capturing emotions during interaction time can help us in making changes on users’ behavior (e.g., changing from stressed to a less stressed state) or even suggesting an user to have a break. This can be all possible if both (1) emotions are captured during interaction and (2) changes are suggested at runtime (e.g., through persuasion). The results of this study suggest that there are significant differences between emotional responses captured during the interaction and those declared at the end.


Emotions Assessment Interaction time Sensors Psychologists Emotional components 


Compliance with ethical standards

Conflict of interest

The authors declare that there is no conflict of interests.


  1. Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25(1):49–59CrossRefGoogle Scholar
  2. Brave S, Nass C (2003) Emotion in human-computer interaction. In: Jacko JA, Sears A (eds) The human-computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, pp 81–96Google Scholar
  3. Chanel G, Kierkels JJ, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Hum Comput Stud 67(8):607–627CrossRefGoogle Scholar
  4. Davidson RJ, Scherer KR, Goldsmith H (2003) Handbook of affective sciences. Oxford University Press, OxfordGoogle Scholar
  5. Desmet P (2004) Funology. In: Blythe MA, Overbeeke K, Monk AF, Wright PC (eds) Measuring emotion: development and application of an instrument to measure emotional responses to products. Kluwer Academic Publishers, Norwell, pp 111–123Google Scholar
  6. Duncker K, Lees LS (1945) On problem-solving. Psychol Monogr 58(5):iCrossRefGoogle Scholar
  7. Ericsson K, Simon H (1993) Protocol analysis; verbal reports as data (revised edition). bradfordbooksGoogle Scholar
  8. Filho GPR, Ueyama J, Villas LA, Pinto AR, Gonalves VP, Pessin G, Pazzi RW, Braun T (2014) Nodepm: a remote monitoring alert system for energy consumption using probabilistic techniques. Sensors 14(1):848–867CrossRefGoogle Scholar
  9. Fontaine JR, Poortinga YH, Setiadi B, Markam SS (2002) Cognitive structure of emotion terms in indonesia and the netherlands. Cogn Emot 16(1):61–86CrossRefGoogle Scholar
  10. Frantzidis CA, Bratsas C, Klados MA, Konstantinidis E, Lithari CD, Vivas AB, Papadelis CL, Kaldoudi E, Pappas C, Bamidis PD (2010) On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications. Inform Technol Biomed IEEE Trans 14(2):309–318CrossRefGoogle Scholar
  11. Gonçalves VP, de Almeida Neris VP, Seraphini S, Dias TCM, Pessin G, Johnson T, Ueyama J (2015) Providing adaptive smartphone interfaces targeted at elderly people: an approach that takes into account diversity among the elderly. Univ Access Inf Soc 1–21. doi: 10.1007/s10209-015-0429-9
  12. Hou X, Liu Y, Sourina O, Tan YRE, Wang L, Mueller-Wittig W (2015) EEG based Stress Monitoring. In: IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2015, pp 3110–3115. doi: 10.1109/SMC.2015.540
  13. Kahou SE, Bouthillier X, Lamblin P, Gulcehre C, Michalski V, Konda K, Jean S, Froumenty P, Dauphin Y, Boulanger-Lewandowski N, Ferrari RC, Mirza M, Warde-Farley D, Courville A, Vincent P, Memisevic R, Pal C, Bengio Y (2015) EmoNets: Multimodal deep learning approaches for emotion recognition in video. J Multimodal User Interfaces 1–13. doi: 10.1007/s12193-015-0195-2
  14. Kukolja D, Popović S, Horvat M, Kovač B, Ćosić K (2014) Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. Int J Hum Comput Stud 72(10):717–727CrossRefGoogle Scholar
  15. Lan Z, Sourina O, Wang LP, Liu Y (2014) Stability of features in real-time EEG-based emotion recognition algorithm. In: International Conference on Cyberworlds (CW), 2014, pp 137–144. doi: 10.1109/CW.2014.27
  16. Lathia N, Rachuri K, Mascolo C, Roussos G (2013) Open source smartphone libraries for computational social science. In: Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, pp 911–920Google Scholar
  17. Li T, An C, Campbell AT, Zhou X (2015) Hilight: Hiding bits in pixel translucency changes. ACM SIGMOBILE Mobile Comput Commun Rev 18(3):62–70CrossRefGoogle Scholar
  18. LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th annual international conference on Mobile systems, applications, and services. ACM. pp 389–402Google Scholar
  19. Mahlke S, Minge M (2008) Consideration of multiple components of emotions in human-technology interaction. In: Peter C, Beale R (eds) Affect and Emotion in Human-Computer Interaction, Lecture Notes in Computer Science, vol 4868. Springer, Berline, pp 51–62Google Scholar
  20. Mano LY, Giancristofaro GT, Faiçal B, Libralon G, Pessin G, Gomes PH, Ueyama J (2015) Exploiting the Use of Ensemble Classifiers to Enhance the Precision of User’s Emotion Classification. In: Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (INNS)Google Scholar
  21. Mitchell TM (1997) Machine learning. McGraw Hill, Boston, MAGoogle Scholar
  22. Monard MC, Baranauskas JA (2003) Conceitos sobre aprendizado de máquina. Sistemas Inteligentes-Fundamentos e Aplicações 1:1Google Scholar
  23. Nahin ANH, Alam JM, Mahmud H, Hasan K (2014) Identifying emotion by keystroke dynamics and text pattern analysis. Behav Inform Technol 33(9):987–996CrossRefGoogle Scholar
  24. Niedenthal PM, Krauth-Gruber S, Ric F (2006) Psychology of emotion: interpersonal, experiential, and cognitive approaches. Psychology Press, New YorkGoogle Scholar
  25. Pedersen C, Togelius J, Yannakakis GN (2010) Modeling player experience for content creation. IEEE Trans Comput Intell AI Games 2(1):54–67CrossRefGoogle Scholar
  26. Peter C, Urban B (2012) Emotion in human-computer interaction. In: Expanding the frontiers of visual analytics and visualization. Springer, pp 239–262Google Scholar
  27. Ramakrishnan S, El Emary IM (2013) Speech emotion recognition approaches in human computer interaction. Telecommun Syst 52(3):1467–1478CrossRefGoogle Scholar
  28. Reijneveld K, de Looze M, Krause F, Desmet P (2003) Measuring the emotions elicited by office chairs. In: Proceedings of the 2003 International Conference on Designing Pleasurable Products and Interfaces, ser. DPPI ’03. ACM, New York. p 6–10Google Scholar
  29. Rosales GCM, Borges de Araujo R, Otsuka JL, da Rocha RV (2011) Using logical sensors network to the accurate monitoring of the learning process in distance education courses. In: Advanced Learning Technologies (ICALT), 2011 11th IEEE International Conference on. IEEE, pp 573–575Google Scholar
  30. Russell J (1980) A circumplex model of affect. J Personal Soc Psychol 39(6):1161–1178CrossRefGoogle Scholar
  31. Russell JA, Weiss A, Mendelsohn GA (1989) Affect grid: a single-item scale of pleasure and arousal. J Personal Soc Psychol 57(3):493CrossRefGoogle Scholar
  32. Santos V, Coca S, Libralon G, Romero R (2013) Imitation of facial expressions for learning emotions in social robotics. In: Proceedings of the 2013 Simposio Brasileiro de Automacao InteligenteGoogle Scholar
  33. Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215MathSciNetCrossRefzbMATHGoogle Scholar
  34. Scherer K (2005) What are emotions? And how can they be measured? Soc Sci Inf 44:695–729CrossRefGoogle Scholar
  35. Van Someren MW, Barnard YF, Sandberg JA et al (1994) The think aloud method: a practical guide to modelling cognitive processes, vol 2. Academic Press, LondonGoogle Scholar
  36. Wang R, Chen F, Chen Z, Li T, Harari G, Tignor S, Zhou X, Ben-Zeev D, Campbell AT (2014) Studentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM. pp 3–14Google Scholar
  37. Wrzus C, Mehl MR (2015) Lab and/or field? Measuring personality processes and their social consequences. Eur J Personal 29(2):250–271CrossRefGoogle Scholar
  38. Xavier RAC, de Almeida Neris VP (2014) Measuring the emotional experience of users through a hybrid semantic approach. In: Proceedings of the 13th Brazilian Symposium on Human Factors in Computing Systems. Sociedade Brasileira de Computação, pp 226–235Google Scholar
  39. Xavier RAC, Garcia FE, de Almeida Neris VP (2012) Decisoes de design de interfaces ruins e o impacto delas na interacao: Um estudo preliminar considerando o estado emocional de idosos. In: Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems, IHC ’12. Brazilian Computer Society, Porto Alegre, Brazil. pp 127–136Google Scholar
  40. Zhou F, Qu X, Helander MG, Jiao JR (2011) Affect prediction from physiological measures via visual stimuli. Int J Hum Comput Stud 69(12):801–819CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Vinícius P. Gonçalves
    • 1
  • Gabriel T. Giancristofaro
    • 1
  • Geraldo P. R. Filho
    • 1
  • Thienne Johnson
    • 2
  • Valéria Carvalho
    • 1
  • Gustavo Pessin
    • 3
  • Vânia P. de Almeida Neris
    • 4
  • Jó Ueyama
    • 1
  1. 1.Institute of Mathematics and Computer Science (ICMC)University of São Paulo (USP)São CarlosBrazil
  2. 2.Department of Computer ScienceThe University of Arizona (UA)TucsonUSA
  3. 3.Institute of Exact and Natural SciencesFederal University of Pará (UFPA) – PABelémBrazil
  4. 4.Department of ComputingFederal University of São Carlos (UFSCar)São CarlosBrazil

Personalised recommendations