User Modeling and User-Adapted Interaction

, Volume 22, Issue 3, pp 255–279 | Cite as

Tune in to your emotions: a robust personalized affective music player

  • Joris H. JanssenEmail author
  • Egon L. van den Broek
  • Joyce H. D. M. Westerink
Open Access
Original paper


The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application.


Mood Music Psychophysiology User modeling Kernel density estimation Validation Affective computing 



We gratefully acknowledge Marjolein van der Zwaag, Tim Tijs, Kathryn Segovia, and Maurits Kaptein for their helpful comments and vivid discussions on an earlier draft of this paper. We also thank three anonymous reviewers and the editor who all provided us detailed feedback on two earlier versions of this paper. Thanks to their comments and suggestions we have been able to revise this article substantially. Finally, we gratefully acknowledge Lynn Packwood for her careful proof reading.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.


  1. Baas M., De Dreu C.K.W., Nijstad B.A.: A meta-analysis of 25 years of moodcreativity research: hedonic tone, activation, or regulatory focus?.  Psychol. Bull. 134, 779–806 (2008)CrossRefGoogle Scholar
  2. Bak A.A., Grobbee D.E.: A randomized study on coffee and blood pressure. J. Hum. Hypertens. 4, 259–264 (1990)Google Scholar
  3. Baumgartner T., Esslen M., Jäncke L.: From emotion perception to emotion experience: emotions evoked by pictures and classical music. Int. J. Psychophysiol. 60(1), 34–43 (2006)CrossRefGoogle Scholar
  4. Beedie C.J., Terry P.C., Lane A.M.: Distinctions between emotion and mood. Cogn. Emot. 19, 847–878 (2005)CrossRefGoogle Scholar
  5. Bishop C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  6. Boehner K., DePaula R., Dourish P., Sengers P.: How emotion is made and measured. Int. J. Hum.-Comput. Stud. 65, 275–291 (2007)CrossRefGoogle Scholar
  7. Boucsein W.: Electrodermal Activity. Plenum Press, New York (1992)Google Scholar
  8. Caberletti, L., Elfmann, K., Kümmel, M., Schierz, C.: Influence of ambient lighting in vehicle interior on the driver’s perception. In: de Kort, Y., IJsselsteijn, W., Vogels, I., Aarts, M., Tenner, A., Smolders, K. (eds.) Proceedings of Experiencing Light 2009 International Conference on the Effects of Light on Wellbeing, pp. 5–13, Eindhoven, The Netherlands (2009)Google Scholar
  9. Cacioppo J., Tassinary L.: Inferring psychological significance from physiological signals. Am. Psychol. 45, 16–28 (1990)CrossRefGoogle Scholar
  10. Calvo R.A., D’Mello S.: Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1, 18–37 (2010)CrossRefGoogle Scholar
  11. Carberry S., de Rosis F.: Introduction to special issue on affective modeling and adaptation. User Model. User-Adapt. Interact. 18, 1–9 (2008)CrossRefGoogle Scholar
  12. Chin N.D.: Emperical evaluation of user models and user adaptive systems. User Model. User-Adapt. Interact. 11, 181–194 (2001)zbMATHCrossRefGoogle Scholar
  13. Clore G.L., Palmer J.: Affective guidance of intelligent agents: how emotion controls cognition. Cogn. Syst. Res. 10(1), 21–30 (2009)CrossRefGoogle Scholar
  14. Csíkszentmihályi M.: Flow: The Psychology of Optimal Experience. Harper Collins, Sussex, UK (1990)Google Scholar
  15. de Rosis F.: Preface: towards adaptation of interaction to affective factors. User Model. User-Adapt. Interact. 11, 267–278 (2001)CrossRefGoogle Scholar
  16. D’Mello S., Craig S.D., Witherspoon A., McDaniel B., Graesser A.: Automatic detection of learners affect from conversational cues. User Model. User-Adapt. Interact. 18, 45–80 (2008)CrossRefGoogle Scholar
  17. Fairclough S.H.: Fundamentals of physiological computing. Interact. Comput. 21, 133–145 (2009)CrossRefGoogle Scholar
  18. Fogg B.J.: Persuas. Technology. Morgan Kaufmann Publishers, New York (2003)Google Scholar
  19. Frijda N.H.: The Emotions. Cambridge University Press, New York (1986)Google Scholar
  20. Geenen R., van de Vijver F.J.R.: A simple test of the law of initial values. Psychophysiology 30(5), 525–530 (1993)CrossRefGoogle Scholar
  21. Gendolla G.H.E.: On the impact of mood on behavior: an integrative theory and a review. Rev. Gen. Psychol. 4, 378–408 (2000)CrossRefGoogle Scholar
  22. Gendolla G.H.E., Brinkman K.: The role of mood states in self-regulation: effects on action preferences and resource mobilization. Eur. Psychol. 10, 187–198 (2005)CrossRefGoogle Scholar
  23. Gendolla G.H.E., Krüsken J.: Mood state and cardiovascular response in active coping with an affect-regulative challenge. Int. J. Psychophysiol. 41, 169–180 (2001)CrossRefGoogle Scholar
  24. Hanson M.A., Powell H.C. Jr, Barth A.T., Ringgenberg K., Calhoun B.H., Aylor J.H. et al.: Body area sensor networks: challenges and opportunities. IEEE Comput. 42, 58–65 (2009)CrossRefGoogle Scholar
  25. Härdle W.: Smoothing Techniques, with Implementations in S. Springer, New York (1991)Google Scholar
  26. Healey, J.A.: Affect detection in the real world: recording and processing physiological signals. In: Proceedings of the IEEE 3rd International Conference on Affective Computing and Intelligent Interaction, ACII, Vol. 1, pp. 729–734. IEEE Press, Amsterdam (2009)Google Scholar
  27. Healey J.A., Picard R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6, 156–166 (2005)CrossRefGoogle Scholar
  28. Healey, J.A., Picard, R.W., Dabek, F.: A new affect-perceiving interface and its application to personalized music selection. In: Turk, M. (ed.) Proceedings of the 1998 Workshop on Perceptual User Interfaces (PUI), San Francisco, CA, USA (1998)Google Scholar
  29. Heinz C., Seeger B.: Cluster kernels: resource-aware kernel density estimators over streaming data. IEEE Trans. Knowl. Data Eng. 20, 880–893 (2008)CrossRefGoogle Scholar
  30. Höök K.: Affective loop experiences: designing for interactional embodiment. Philos. Trans. R. Soc. B 364, 3585–3595 (2009)CrossRefGoogle Scholar
  31. Husain G., Thompson W.F., Schellenberg E.G.: Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Percept. 20, 151–171 (2002)CrossRefGoogle Scholar
  32. Janssen J.H., Bailenson J.N., IJsselstein W.A., Westerink J.H.D.M.: Intimate heartbeats: opportunities for affective communication technology. IEEE Trans. Affect. Comput. 1(2), 72–80 (2010)CrossRefGoogle Scholar
  33. Kaptein M., Eckles D.: Selecting effective means to any end: futures and ethics of per30 suasion profiling. In: Ploug, T., Hasle, P., Oinas-Kukkonen, H. (eds.) Persuasive Technology, pp. 82–93. Springer, Berlin (2010)CrossRefGoogle Scholar
  34. Kim J., André E.: Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2067–2083 (2008)CrossRefGoogle Scholar
  35. Kowal J., Fortier M.: Motivational determinants of flow: contributions from selfdetermination theory. J. Soc. Psychol. 139, 355–368 (1999)CrossRefGoogle Scholar
  36. Lemmens, P., Crompvoets, F., Brokken, D., van den Eerenbeemd, J., De Vries, G.-J.: A body-conforming tactile jacket to enrich movie viewing. In World Haptics Conference, pp. 7–12. IEEE, Los Alamitos (2009)Google Scholar
  37. Lesiuk T.: The effect of music listening on work performance. Psychol. Music 33, 173–191 (2005)CrossRefGoogle Scholar
  38. Liljedahl, M., Sjömark, C., Lefford, N.: Using music to promote physical well-being via computer-mediated interaction. In MusicNetwork Open Workshop, 5 (2005)Google Scholar
  39. Mandel M., Poliner G., Ellis D.: Support vector machine active learning for music retrieval. ACM Multimed. Syst. J. 12, 3–13 (2006)CrossRefGoogle Scholar
  40. Matthews G., Jones D.M., Chamberlain A.G.: Refining the measurement of mood: the UWIST mood adjective checklist. Br. J. Psychol. 81, 17–42 (1990)CrossRefGoogle Scholar
  41. McFarland R.A., Kennison R.: Asymmetry in the relationship between finger temperature changes and emotional state in males. Appl. Psychophysiol. Biofeedback 14(4), 281–290 (1989)Google Scholar
  42. North A.C.H., David J.: Musical preferences during and after relaxation and exercise. Am. J. Psychol. 113, 43–67 (2000)CrossRefGoogle Scholar
  43. North A.C.H., Hargreaves D.J., Hargreaves J.J.: Uses of music in everyday life. Music Percept. 22, 41–77 (2004)CrossRefGoogle Scholar
  44. Oliver, N., Flores-Mangas, F.: MPTrain: a mobile, music and physiology-based personal trainer. In: Proceedings of the 8th Conference on Human–Computer Interaction with Mobile Devices and Services, pp. 21–28. ACM, New York (2006)Google Scholar
  45. Oliver, N., Kregor-Stickles, L.: PAPA: physiology and purpose-aware automatic playlist generation. In: Lemström, K., Tindale, A., Dannenberg, R. (eds.) Proceedings of the 7th International Conference on Music Information Retrieval (ISMIR), pp. 250–253, Victoria, Canada, 8–12 October 2006Google Scholar
  46. Ophira E., Nass C., Wagner A.D.: Cognitive control in media multitaskers. Proc. Natl Acad. Sci. 106, 15583–15587 (2009)CrossRefGoogle Scholar
  47. Pantic M., Patras I.: Dynamics of facial expressions: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Man Syst. Cybernet. B 36, 433–449 (2006)CrossRefGoogle Scholar
  48. Pelletier C.L.: The effect of music on decreasing arousal due to stress: a meta-analysis. J. Music Ther. 41, 192–214 (2004)Google Scholar
  49. Peter C., Herbon A.: Emotion representation and physiology assignments in digital systems. Interact. Comput. 18, 139–170 (2006)CrossRefGoogle Scholar
  50. Picard R.W.: Affective Computing. MIT Press, Cambridge (1997)Google Scholar
  51. Picard R.W.: Affective computing: challenges. Int. J. Hum.-Comput. Stud. 59, 55–64 (2003)CrossRefGoogle Scholar
  52. Prinz J.J.: Gut Reactions: A Perceptual Theory of Emotion. Oxford University Press, New York (2004)Google Scholar
  53. Rentfrow P.J., Gosling S.D.: The do re mi’s of everyday life: the structure and personality correlates of music preference. J. Pers. Soc. Psychol. 84, 1236–1256 (2003)CrossRefGoogle Scholar
  54. Rickard N.S.: Intense emotional responses to music: A test of the physiological arousal hypothesis. Psychol. Music 32, 371–388 (2004)CrossRefGoogle Scholar
  55. Rimm-Kaufman S.E., Kagan J.: The psychological significance of changes in skin temperature. Motiv. Emot. 20(1), 64–78 (1996)CrossRefGoogle Scholar
  56. Ritossa D.A., Rickard N.S.: The relative utility of ‘pleasantness’ and ‘liking’ dimensions in predicting the emotions expressed by music. Psychol. Music 32, 5–22 (2004)CrossRefGoogle Scholar
  57. Russell J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110, 145–172 (2003)CrossRefGoogle Scholar
  58. Rusting C.L.: Personality, mood, and cognitive processing of emotional information: three conceptual frameworks. Psychol. Bull. 124, 165–196 (1998)CrossRefGoogle Scholar
  59. Saarikallio S., Erkkilä J.: Role of music in adolescents’ mood regulation. Psychol. Music 35, 88–109 (2007)CrossRefGoogle Scholar
  60. Scott D., Sain S.: Multidimensional density estimation. In: Rao, C.R., Wegman, E.J., Solka, J.L. (eds.) Handbook of Statistics, Vol. 24, pp. 229–261. Elsevier, North Holland (2005)Google Scholar
  61. Silverman B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall, London (1986)zbMATHGoogle Scholar
  62. Sloboda J.A.: Exploring the Musical Mind: Cognition, Emotion, Ability, Function. Oxford University Press, New York (2005)Google Scholar
  63. Sotiropoulos D.N., Lampropoulos A.S., Tsihrintzis G.A.: MUSIPER: a system for modeling music similarity perception based on objective feature subset selection. User Model. User-Adapt. Interact. 18, 315–348 (2008)CrossRefGoogle Scholar
  64. Thayer R.E.: The Biopsychology of Mood and Activation. Oxford University Press, New York (1989)Google Scholar
  65. Tractinsky N.: Tools over solutions? comments on interacting with computers special issue on affective computing. Interact. Comput. 16, 751–757 (2004)CrossRefGoogle Scholar
  66. Turlach, B.A.: Bandwidth selection in kernel density estimation: a review. Discussion Paper 9317, Institut de Statistique, Voie du Roman Pays 34, B-1348 Louvain-la-Neuve (1993)Google Scholar
  67. Vaillant G.: Aging Well: Surprising Guideposts to a Happier Life from the Landmark Harvard Study of Adult Development. Little, Brown and Company, Boston (2003)Google Scholar
  68. Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M.: Guidelines for Affective Signal Processing (ASP): From lab to life. In Proceedings of the IEEE 3rd international conference on affective computing and intelligent interaction, ACII, Vol. 1, pp. 704–709. IEEE Press, Amsterdam, The Netherlands (2009a)Google Scholar
  69. Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M., Healey, J.A.: Prerequisites for Affective Signal Processing (ASP). In: Encarnaçã, P., Veloso, A. (eds.) Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, pp. 426–433, Porto, Portugal (2009b)Google Scholar
  70. Van den Broek E.L., Lisý V., Janssen J.H., Westerink J.H.D.M., Schut M.H., Tuinenbreijer K.: Affective man–machine interface: unveiling human emotions through biosignals. In: Fred, A., Filipe, J., Gamboa, H. (eds.) Biomedical Engineering Systems and Technologies: BIOSTEC2009 Selected Revised Papers, Vol. 52., pp. 21–47. Springer, Berlin (2010)Google Scholar
  71. Wagenaar W.A.: Note on the construction of digram-balanced latin squares. Psychol. Bull. 72, 384–386 (1969)CrossRefGoogle Scholar
  72. Webster G.D., Weir C.G.: Emotional responses to music: interactive effects of mode, texture, and tempo. Motiv. Emot. 29, 19–39 (2005)CrossRefGoogle Scholar
  73. Westerink, J.H.D.M., De Vries, G., Waele, S., Eerenbeemd, J., Boven, M., Ouwerkerk, M.: Emotion measurement platform for daily life situations. In: Nijholt, A., Cohn, J., Pantic, M. (eds.) Proceedings of ACII’09: Affective Computing and Intelligent Interaction, pp. 217–223. IEEE, Los Alamitos (2009)Google Scholar
  74. Wilder J.: Stimulus and Response: The Law of Initial Values. Wright, Bristol (1967)Google Scholar
  75. Wilhelm P., Schoebi D.: Assessing mood in daily life: structural validity, sensitivity to change, and reliability of a short-scale to measure three basic dimensions of mood. Eur. J. Psychol. Assess. 23, 258–267 (2007)CrossRefGoogle Scholar
  76. Wilson G.F., Russell C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45, 635–644 (2003)CrossRefGoogle Scholar
  77. Yannakakis G.N., Hallam J., Lund H.H.: Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapt. Interact. 18, 207–243 (2008)CrossRefGoogle Scholar
  78. Zeng Z., Pantic M., Roisman G.I., Huang T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)CrossRefGoogle Scholar

Copyright information

© The Author(s) 2011

Authors and Affiliations

  • Joris H. Janssen
    • 1
    • 2
    Email author
  • Egon L. van den Broek
    • 3
    • 4
  • Joyce H. D. M. Westerink
    • 2
  1. 1.Human Technology Interaction, Department of Industrial Engineering and Innovation SciencesEindhoven University of TechnologyEindhovenThe Netherlands
  2. 2.Department of Brain, Body, and BehaviorPhilips ResearchEindhovenThe Netherlands
  3. 3.Human Media Interaction, Faculty of Electrical Engineering, Mathematics, and Computer ScienceUniversity of TwenteEnschedeThe Netherlands
  4. 4.Karakter U.C., Radboud University Medical Center NijmegenNijmegenThe Netherlands

Personalised recommendations