Perso2U: Exploration of User Emotional States to Drive Interface Adaptation

  • Julián Andrés GalindoEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 895)


Taking into account dynamic user properties such as emotions for interfaces adaptation at runtime is a challenging task. To deal with this issue, we propose to personalize user interfaces at runtime based on user’s emotions. This approach depends on emotion recognition tools to allow an Inferring Engine to deduce user emotions during the interaction. However, this inference releases many emotions without aggregating them. It makes more difficult the interpretation of user experience. Thus, we explore the feasibility of inferring similar emotional states (negative, positive and neutral) by grouping individual emotions. To achieve our goal, this paper reports on the results of an experiment to compare detected emotional states from different face recognition tools in web interaction. It evidences that it is feasible to infer similar emotion states (positive, negative, and neutral) from different emotion recognition tools, and the level of this similarity is still premature to have a robust categorization.


User interface adaptation Inferring engine Emotion recognition Face detection Runtime 


  1. 1.
    Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interact. Comput. 15(3), 289–308 (2003)Google Scholar
  2. 2.
    Reeves, B., Nass, C.: How people treat computers, television, and new media like real people and places. CSLI Publications and Cambridge University Press Cambridge, UK (1996)Google Scholar
  3. 3.
    Carberry, S., de Rosis, F.: Introduction to special Issue on ‘affective modeling and adaptation’. User Model. User-Adapt. Interact. 18(1–2), 1–9 (2008)Google Scholar
  4. 4.
    Cyr, D.: Emotion and website design. In: Soegaard, M., Dam, R.F. (eds.) The Encyclopedia of Human-Computer Interaction. The Interaction-Design Foundation, Aarhus (2013)Google Scholar
  5. 5.
    Nasoz, F.: Adaptive Intelligent User Interfaces With Emotion Recognition. University of Central Florida Orlando, Florida (2004)Google Scholar
  6. 6.
    Hudlicka, E., McNeese, M.D.: Assessment of user affective and belief states for interface adaptation: application to an air force pilot task. User Model. User-Adapt. Interact. 12(1), 1–47 (2002)Google Scholar
  7. 7.
    Martins, C., Faria, L., De Carvalho, C.V., Carrapatoso, E.: User modeling in adaptive hypermedia educational systems. Educ. Technol. Soc. 11(1), 194–207 (2008)Google Scholar
  8. 8.
    de Rosis, F., Novielli, N., Carofiglio, V., Cavalluzzi, A., Carolis, B.D.: User modeling and adaptation in health promotion dialogs with an animated character. J. Biomed. Inform. 39(5), 514–531 (2006)Google Scholar
  9. 9.
    Rowe, J., Mott, B., McQuiggan, S., Robison, J., Lee, S., Lester, J.: Crystal island: a narrative-centered learning environment for eighth grade microbiology. In: Workshop on Intelligent Educational Games at the 14th International Conference on Artificial Intelligence in Education, pp. 11–20, Brighton, UK (2009)Google Scholar
  10. 10.
    Forbes-Riley, K., Litman, D.: Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Comput. Speech Lang. 25(1), 105–126 (2011)Google Scholar
  11. 11.
    Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User-Adapt. Interact. 18(1–2), 125–173 (2008)Google Scholar
  12. 12.
    Graesser, A.C., et al.: The relationship between affective states and dialog patterns during interactions with AutoTutor. J. Interact. Learn. Res. 19(2), 293 (2008)Google Scholar
  13. 13.
    Meudt, S., et al.: Going further in affective computing: how emotion recognition can improve adaptive user interaction. In: Esposito, A., Jain, L.C. (eds.) Toward Robotic Socially Believable Behaving Systems, vol. 105, pp. 73–103. Springer International Publishing, Cham (2016)Google Scholar
  14. 14.
    Märtin, C., Rashid, S., Herdin, C.: Designing responsive interactive applications by emotion-tracking and pattern-based dynamic user interface adaptation. In: Kurosu, M. (ed.) HCI 2016, Part III. LNCS, vol. 9733, pp. 28–36. Springer, Cham (2016). Scholar
  15. 15.
    Hudlicka, E.: Increaing sia architecture realism by modeling and adapting to affect and personality. In: Dautenhahn, K., Bond, A., Cañamero, L., Edmonds, B. (eds.) Socially Intelligent Agents. Multiagent Systems, Artificial Societies, and Simulated Organizations, pp. 53–60. Springer, Boston (2002). Scholar
  16. 16.
    Plutchik, R., Kellerman, H.: Theories of Emotion, vol. 1 of Emotion: Theory, Research, and Experience. Academic Press, New York (1980)Google Scholar
  17. 17.
    Scherer, K.R.: What are emotions? and how can they be measured? Soc. Sci. Inf. 44(4), 695–729 (2005)Google Scholar
  18. 18.
    Fischer, G.: User modeling in human–computer interaction. User Model. User-Adapt. Interact. 11(1–2), 65–86 (2001)Google Scholar
  19. 19.
    Russell, J.A., Mehrabian, A.: Evidence for a three-factor theory of emotions. J. Res. Pers. 11(3), 273–294 (1977)Google Scholar
  20. 20.
    Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)Google Scholar
  21. 21.
    Märtin, C., Herdin, C., Engel, J.: Model-based user-interface adaptation by exploiting situations, emotions and software patterns. In: International Conference on Computer-Human Interaction Research and Applications (2017)Google Scholar
  22. 22.
    Galindo, J., Dupuy-Chessa, S., Céret, E.: Toward a UI adaptation approach driven by user emotions. In: Presented at the ACHI07, Nice, France (2017)Google Scholar
  23. 23.
    Céret, E., Dupuy-Chessa, S., Calvary, G., Bittar, M.: System and method for magnetic adaptation of a user interface. TPI2015053 déposé via France INPI le 7 juillet 2015, 2ème dépôt le 7 juillet (2016)Google Scholar
  24. 24.
    Dupuy-Chessa, S., Laurillau, Y., Céret, E.: Considering aesthetics and usability temporalities in a model based development process. In: Actes de la 28ième conférence francophone sur l’Interaction Homme-Machine, pp. 25–35 (2016)Google Scholar
  25. 25.
    Posner, J., Russell, J.A., Peterson, B.S.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(3), 715–734 (2005)Google Scholar
  26. 26.
    Steunebrink, B.R., et al.: The Logical structure of emotions (2010)Google Scholar
  27. 27.
    Pearson, K.X.: On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Lond. Edinb. Dublin Philos. Mag. J. Sci. 50(302), 157–175 (1900)Google Scholar
  28. 28.
    Karl Pearson, F.R.S.: Mathematical Contributions to the Theory of Evolution. Dulau Co, London (1904)Google Scholar
  29. 29.
    Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37–46 (1960)Google Scholar
  30. 30.
    Hume, D.: Emotions and moods. In: Robbins, S.P., Judge, T.A. (eds.) Organizational Behavior, pp. 258–297. Pearson Prentice Hall, Upper Saddle River (2012)Google Scholar
  31. 31.
    Larsen, R.J., Diener, E.: Affect intensity as an individual difference characteristic: a review. J. Res. Pers. 21(1), 1–39 (1987)Google Scholar
  32. 32.
    Diener, E., Sandvik, E., Larsen, R.J.: Age and sex effects for emotional intensity. Dev. Psychol. 21(3), 542 (1985)Google Scholar
  33. 33.
    Watson, D.: Mood and Temperament. Guilford Press, New York (2000)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Departamento de Informática y Ciencias de la ComputaciónEscuela Politécnica NacionalQuitoEcuador
  2. 2.University Grenoble Alpes, CNRS, LIGGrenobleFrance

Personalised recommendations