Modeling Aspects in Human-Computer Interaction: Adaptivity, User Characteristics and Evaluation

  • Tatiana Gossen
  • Ingo Siegert
  • Andreas Nürnberger
  • Kim Hartmann
  • Michael Kotzyba
  • Andreas Wendemuth
Part of the Cognitive Technologies book series (COGTECH)


During system interaction, the user’s emotions and intentions shall be adequately determined and predicted to recognize tendencies in his or her interests and dispositions. This allows for the design of an evolving search user interface (ESUI) which adapts to changes in the user’s emotional reaction and the users’ needs and claims.

Here, we concentrate on the front end of the search engine and present two prototypes, one which can be customised to the user’s needs and one that takes the user’s age as a parameter to roughly approximate the user’s skill space and for subsequent system adaptation. Further, backend algorithms to detect the user’s abilities are required in order to have an adaptive system.

To develop an ESUI, user studies with users of gradually different skills have been conducted with groups of young users. In order to adapt the interaction dialog, we propose monitoring the user’s emotional state. This enables monitoring early detection of the user’s problems in interacting with the system, and allows us to adapt the dialog to get the user on the right path. Therefore, we investigate methods to detect changes in the user’s emotional state.

We furthermore propose a user mood modeling from a technical perspective based on a mechanical spring model in PAD-space, which is able to incorporate several psychological observations. This implementation has the advantage of only three internal parameters and one user-specific parameter-pair.

We present a technical implementation of that model in our system and evaluate the principal function of the proposed model on two different databases. Especially on the EmoRecWoz corpus, we were able to show that the generated mood course matched the experimental setting.

By utilizing the user-specific parameter-pair the personality trait extraversion was modeled. This trait is supposed to regulate the individual emotional experiences.

Technically, we present an implementable feature-based, dimensional model for emotion analysis which is able to track and predict the temporal development of emotional reactions in an evolving search user interface, and which is adjustable based on mood and personality traits.



This work was done within the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).


  1. 1.
    Amstad, T.: Wie verständlich sind unsere Zeitungen? Ph.D. Thesis, University of Zurich (1978)Google Scholar
  2. 2.
    Arapakis, I., Athanasakos, K., Jose, J.M.: A comparison of general vs personalised affective models for the prediction of topical relevance. In: Proceedings of the 33rd ACM SIGIR’10, pp. 371–378 (2010)Google Scholar
  3. 3.
    Aula, A.: User study on older adults’use of the web and search engines. Univ. Access Inf. Soc. 4(1), 67–81 (2005)CrossRefGoogle Scholar
  4. 4.
    Aula, A., Käki, M.: Less is more in web search interfaces for older adults. First Monday 10(7) (2005)Google Scholar
  5. 5.
    Becker-Asano, C.: WASABI: affect simulation for agents with believable interactivity. Ph.D. Thesis, University of Bielefeld (2008)Google Scholar
  6. 6.
    Böck, R., Limbrecht, K., Walter, S., Hrabal, D., Traue, H.C., Glüge, S., Wendemuth, A.: Intraindividual and interindividual multimodal emotion analyses in human-machine-interaction. In: Proceedings of the IEEE CogSIMA, New Orleans, pp. 59–64 (2012)Google Scholar
  7. 7.
    Böck, R., Limbrecht-Ecklundt, K., Siegert, I., Walter, S., Wendemuth, A.: Audio-based pre-classification for semi-automatic facial expression coding. In: Kurosu, M. (ed.) Human-Computer Interaction. Towards Intelligent and Implicit Interaction. Lecture Notes in Computer Science, vol. 8008, pp. 301–309. Springer, Cham (2013)CrossRefGoogle Scholar
  8. 8.
    Carpenter, S.M., Peters, E., Västfjäll, D., Isen, A.M.: Positive feelings facilitate working memory and complex decision making among older adults. Cognit. Emot. 27, 184–192 (2013)CrossRefGoogle Scholar
  9. 9.
    Costa, P.T., McCrae, R.R.: The NEO Personality Inventory Manual. Psychological Assessment Resources, Odessa (1985)Google Scholar
  10. 10.
    Crawford, J.R., Henry, J.D.: The positive and negative affect schedule (PANAS): construct validity, measurement properties and normative data in a large non-clinical sample. Br. J. Clin. Psychol. 43, 245–265 (2004)CrossRefGoogle Scholar
  11. 11.
    Eickhoff, C., Azzopardi, L., Hiemstra, D., De Jong, F., de Vries, A., Dowie, D., Duarte, S., Glassey, R., Gyllstrom, K., Kruisinga, F., Marshall, K., Moens, S., Polajnar, T., van der Sluis, F.: Emse: initial evaluation of a child-friendly medical search system. In: Proceedings of the 4th ACM Information Interaction in Context Symposium, pp. 282–285 (2012)Google Scholar
  12. 12.
    Ellsworth, P., Scherer, K.: Appraisal Processes in Emotion, pp. 572–595. Oxford University Press, New York (2003)Google Scholar
  13. 13.
    Gebhard, P.: ALMA a layered model of affect. In: Proceedings of the 4th ACM AAMAS, Utrecht, pp. 29–36 (2005)Google Scholar
  14. 14.
    Gossen, T., Nürnberger, A.: Specifics of information retrieval for young users: a survey. Inf. Process. Manag. 49(4), 739–756 (2013)CrossRefGoogle Scholar
  15. 15.
    Gossen, T., Nitsche, M., Nürnberger, A.: Knowledge journey: a web search interface for young users. In: Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval (HCIR’12), pp. 1:1–1:10. ACM, Cambridge (2012)Google Scholar
  16. 16.
    Gossen, T., Nitsche, M., Nürnberger, A.: Evolving search user interfaces. In: Proceedings of EuroHCIR 2013 Workshop. Dublin, Ireland (2013)Google Scholar
  17. 17.
    Gossen, T., Nitsche, M., Vos, J., Nürnberger, A.: Adaptation of a search user interface towards user needs - a prototype study with children & adults. In: Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval (HCIR’13), Vancouver, BC (2013)Google Scholar
  18. 18.
    Gossen, T., Kotzyba, M., Nürnberger, A.: Knowledge journey exhibit: towards age-adaptive search user interfaces. In: Hanbury, A., Kazai, G., Rauber, A., Fuhr, N. (eds.) Advances in Information Retrieval. Lecture Notes in Computer Science, vol. 9022, pp. 781–784. Springer, Cham (2015)Google Scholar
  19. 19.
    Hartmann, K., Siegert, I., Glüge, S., Wendemuth, A., Kotzyba, M., Deml, B.: Describing human emotions through mathematical modelling. In: Proceedings of the 7th MATHMOD, Vienna, pp. 463–468 (2012)Google Scholar
  20. 20.
    Hearst, M.: Search User Interfaces. Cambridge University Press, Cambridge (2009)CrossRefGoogle Scholar
  21. 21.
    Hutchinson, H., Druin, A., Bederson, B., Reuter, K., Rose, A., Weeks, A.: How do I find blue books about dogs? The errors and frustrations of young digital library users. In: Proceedings of the 11th International Conference on Human-Computer Interaction (2005)Google Scholar
  22. 22.
    Jansen, M., Bos, W., van der Vet, P., Huibers, T., Hiemstra, D.: TeddIR: tangible information retrieval for children. In: Proceedings of the 9th ACM International conference on Interaction Design and Children, Barcelona, 2010, pp. 282–285Google Scholar
  23. 23.
    Kotzyba, M., Deml, B., Neumann, H., Glüge, S., Hartmann, K., Siegert, I., Wendemuth, A., Traue, H.C., Walter, S.: Emotion detection by event evaluation using fuzzy sets as appraisal variables. In: Proceedings of the 11th ICCM, Berlin, pp. 123–124 (2012)Google Scholar
  24. 24.
    Larsen, R.J., Fredrickson, B.L.: Measurement Issues in Emotion Research, pp. 40–60. Russell Sage Foundation, New York (1999)Google Scholar
  25. 25.
    Larsen, R.J., Ketelaar, T.: Personality and susceptibility to positive and negative emotional states. J. Pers. Soc. Psychol. 61, 132–140 (1991)CrossRefGoogle Scholar
  26. 26.
    Morris, J.D.: SAM: the self-assessment manikin an efficient cross-cultural measurement of emotional response. J. Advert. Res. 35, 63–68 (1995)Google Scholar
  27. 27.
    Morris, W.N.: Mood: The Frame of Mind. Springer, New York (1989)CrossRefGoogle Scholar
  28. 28.
    Moshfeghi, Y.: Role of emotion in information retrieval. Ph.D. Thesis, University of Glasgow, School of Computing Science (2012)Google Scholar
  29. 29.
    Moshfeghi, Y., Piwowarski, B., Jose, J.: Handling data sparsity in collaborative filtering using emotion and semantic based features. In: Proceedings of the 34th ACM SIGIR’11, pp. 625–634 (2011)Google Scholar
  30. 30.
    Nolen-Hoeksema, S., Fredrickson, B., Loftus, G., Wagenaar, W.: Atkinson & Hilgard’s Introduction to Psychology, 15 edn. Cengage Learning EMEA, Hampshire (2009)Google Scholar
  31. 31.
    Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1990)Google Scholar
  32. 32.
    Scherer, K.R.: Appraisal Considered as a Process of Multilevel Sequential Checking, pp. 92–120. Oxford University Press, Oxford (2001)Google Scholar
  33. 33.
    Siegert, I.: Emotional and user-specific cues for improved analysis of naturalistic interactions. Ph.D. Thesis, Otto von Guericke University Magdeburg (2015)Google Scholar
  34. 34.
    Siegert, I., Böck, R., Wendemuth, A.: The influence of context knowledge for multimodal annotation on natural material. In: Joint Proceedings of the IVA 2012 Workshops, Santa Cruz, pp. 25–32 (2012)Google Scholar
  35. 35.
    Siegert, I., Böck, R., Wendemuth, A.: Modeling users’ mood state to improve human-machine-interaction. In: Esposito, A., Esposito, A.M., Vinciarelli, A., Hoffmann, R., Müller, V.C. (eds.) Cognitive Behavioural Systems. Lecture Notes in Computer Science, vol. 7403, pp. 273–279. Springer, Berlin (2012)CrossRefGoogle Scholar
  36. 36.
    Siegert, I., Hartmann, K., Glüge, S., Wendemuth, A.: Modelling of emotional development within human-computer-interaction. Kognitive Systeme 1 s.p. (2013)Google Scholar
  37. 37.
    Siegert, I., Böck, R., Wendemuth, A.: Inter-rater reliability for emotion annotation in human-computer interaction – comparison and methodological improvements. J. Multimodal User Interfaces 8, 17–28 (2014)CrossRefGoogle Scholar
  38. 38.
    Steichen, B., Ashman, H., Wade, V.: A comparative survey of personalised information retrieval and adaptive hypermedia techniques. Inf. Process. Manag. 48(4), 698–724 (2012)CrossRefGoogle Scholar
  39. 39.
    Stober, S., Nürnberger, A.: Adaptive music retrieval–a state of the art. Multimed. Tools Appl. 65, 467–494 (2013)CrossRefGoogle Scholar
  40. 40.
    Truong, K.P., Neerincx, M.A., Leeuwen, D.A.V.: Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data. In: Proceedings of the INTERSPEECH-2008, Brisbane, pp. 318–321 (2008)Google Scholar
  41. 41.
    Walter, S., Scherer, S., Schels, M., Glodek, M., Hrabal, D., Schmidt, M., Böck, R., Limbrecht, K., Traue, H.C., Schwenker, F.: Multimodal emotion classification in naturalistic user behavior. In: Jacko, J. (ed.) Human-Computer Interaction. Towards Mobile and Intelligent Interaction Environments. Lecture Notes in Computer Science, vol. 6763, pp. 603–611. Springer, Cham (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Tatiana Gossen
    • 1
  • Ingo Siegert
    • 2
  • Andreas Nürnberger
    • 1
  • Kim Hartmann
    • 2
  • Michael Kotzyba
    • 1
  • Andreas Wendemuth
    • 3
  1. 1.Data & Knowledge Engineering GroupOtto von Guericke UniversityMagdeburgGermany
  2. 2.Cognitive Systems GroupOtto von Guericke UniversityMagdeburgGermany
  3. 3.Center for Behavioral Brain SciencesMagdeburgGermany

Personalised recommendations