Skip to main content
Download PDF

Tune in to your emotions: a robust personalized affective music player

User Modeling and User-Adapted Interaction volume 22, pages 255–279 (2012)Cite this article

Abstract

The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application.

References

  • Baas M., De Dreu C.K.W., Nijstad B.A.: A meta-analysis of 25 years of moodcreativity research: hedonic tone, activation, or regulatory focus?.  Psychol. Bull. 134, 779–806 (2008)

    Article  Google Scholar 

  • Bak A.A., Grobbee D.E.: A randomized study on coffee and blood pressure. J. Hum. Hypertens. 4, 259–264 (1990)

    Google Scholar 

  • Baumgartner T., Esslen M., Jäncke L.: From emotion perception to emotion experience: emotions evoked by pictures and classical music. Int. J. Psychophysiol. 60(1), 34–43 (2006)

    Article  Google Scholar 

  • Beedie C.J., Terry P.C., Lane A.M.: Distinctions between emotion and mood. Cogn. Emot. 19, 847–878 (2005)

    Article  Google Scholar 

  • Bishop C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)

    MATH  Google Scholar 

  • Boehner K., DePaula R., Dourish P., Sengers P.: How emotion is made and measured. Int. J. Hum.-Comput. Stud. 65, 275–291 (2007)

    Article  Google Scholar 

  • Boucsein W.: Electrodermal Activity. Plenum Press, New York (1992)

    Google Scholar 

  • Caberletti, L., Elfmann, K., Kümmel, M., Schierz, C.: Influence of ambient lighting in vehicle interior on the driver’s perception. In: de Kort, Y., IJsselsteijn, W., Vogels, I., Aarts, M., Tenner, A., Smolders, K. (eds.) Proceedings of Experiencing Light 2009 International Conference on the Effects of Light on Wellbeing, pp. 5–13, Eindhoven, The Netherlands (2009)

  • Cacioppo J., Tassinary L.: Inferring psychological significance from physiological signals. Am. Psychol. 45, 16–28 (1990)

    Article  Google Scholar 

  • Calvo R.A., D’Mello S.: Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1, 18–37 (2010)

    Article  Google Scholar 

  • Carberry S., de Rosis F.: Introduction to special issue on affective modeling and adaptation. User Model. User-Adapt. Interact. 18, 1–9 (2008)

    Article  Google Scholar 

  • Chin N.D.: Emperical evaluation of user models and user adaptive systems. User Model. User-Adapt. Interact. 11, 181–194 (2001)

    MATH  Article  Google Scholar 

  • Clore G.L., Palmer J.: Affective guidance of intelligent agents: how emotion controls cognition. Cogn. Syst. Res. 10(1), 21–30 (2009)

    Article  Google Scholar 

  • Csíkszentmihályi M.: Flow: The Psychology of Optimal Experience. Harper Collins, Sussex, UK (1990)

    Google Scholar 

  • de Rosis F.: Preface: towards adaptation of interaction to affective factors. User Model. User-Adapt. Interact. 11, 267–278 (2001)

    Article  Google Scholar 

  • D’Mello S., Craig S.D., Witherspoon A., McDaniel B., Graesser A.: Automatic detection of learners affect from conversational cues. User Model. User-Adapt. Interact. 18, 45–80 (2008)

    Article  Google Scholar 

  • Fairclough S.H.: Fundamentals of physiological computing. Interact. Comput. 21, 133–145 (2009)

    Article  Google Scholar 

  • Fogg B.J.: Persuas. Technology. Morgan Kaufmann Publishers, New York (2003)

    Google Scholar 

  • Frijda N.H.: The Emotions. Cambridge University Press, New York (1986)

    Google Scholar 

  • Geenen R., van de Vijver F.J.R.: A simple test of the law of initial values. Psychophysiology 30(5), 525–530 (1993)

    Article  Google Scholar 

  • Gendolla G.H.E.: On the impact of mood on behavior: an integrative theory and a review. Rev. Gen. Psychol. 4, 378–408 (2000)

    Article  Google Scholar 

  • Gendolla G.H.E., Brinkman K.: The role of mood states in self-regulation: effects on action preferences and resource mobilization. Eur. Psychol. 10, 187–198 (2005)

    Article  Google Scholar 

  • Gendolla G.H.E., Krüsken J.: Mood state and cardiovascular response in active coping with an affect-regulative challenge. Int. J. Psychophysiol. 41, 169–180 (2001)

    Article  Google Scholar 

  • Hanson M.A., Powell H.C. Jr, Barth A.T., Ringgenberg K., Calhoun B.H., Aylor J.H. et al.: Body area sensor networks: challenges and opportunities. IEEE Comput. 42, 58–65 (2009)

    Article  Google Scholar 

  • Härdle W.: Smoothing Techniques, with Implementations in S. Springer, New York (1991)

    Google Scholar 

  • Healey, J.A.: Affect detection in the real world: recording and processing physiological signals. In: Proceedings of the IEEE 3rd International Conference on Affective Computing and Intelligent Interaction, ACII, Vol. 1, pp. 729–734. IEEE Press, Amsterdam (2009)

  • Healey J.A., Picard R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6, 156–166 (2005)

    Article  Google Scholar 

  • Healey, J.A., Picard, R.W., Dabek, F.: A new affect-perceiving interface and its application to personalized music selection. In: Turk, M. (ed.) Proceedings of the 1998 Workshop on Perceptual User Interfaces (PUI), San Francisco, CA, USA (1998)

  • Heinz C., Seeger B.: Cluster kernels: resource-aware kernel density estimators over streaming data. IEEE Trans. Knowl. Data Eng. 20, 880–893 (2008)

    Article  Google Scholar 

  • Höök K.: Affective loop experiences: designing for interactional embodiment. Philos. Trans. R. Soc. B 364, 3585–3595 (2009)

    Article  Google Scholar 

  • Husain G., Thompson W.F., Schellenberg E.G.: Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Percept. 20, 151–171 (2002)

    Article  Google Scholar 

  • Janssen J.H., Bailenson J.N., IJsselstein W.A., Westerink J.H.D.M.: Intimate heartbeats: opportunities for affective communication technology. IEEE Trans. Affect. Comput. 1(2), 72–80 (2010)

    Article  Google Scholar 

  • Kaptein M., Eckles D.: Selecting effective means to any end: futures and ethics of per30 suasion profiling. In: Ploug, T., Hasle, P., Oinas-Kukkonen, H. (eds.) Persuasive Technology, pp. 82–93. Springer, Berlin (2010)

    Chapter  Google Scholar 

  • Kim J., André E.: Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2067–2083 (2008)

    Article  Google Scholar 

  • Kowal J., Fortier M.: Motivational determinants of flow: contributions from selfdetermination theory. J. Soc. Psychol. 139, 355–368 (1999)

    Article  Google Scholar 

  • Lemmens, P., Crompvoets, F., Brokken, D., van den Eerenbeemd, J., De Vries, G.-J.: A body-conforming tactile jacket to enrich movie viewing. In World Haptics Conference, pp. 7–12. IEEE, Los Alamitos (2009)

  • Lesiuk T.: The effect of music listening on work performance. Psychol. Music 33, 173–191 (2005)

    Article  Google Scholar 

  • Liljedahl, M., Sjömark, C., Lefford, N.: Using music to promote physical well-being via computer-mediated interaction. In MusicNetwork Open Workshop, 5 (2005)

  • Mandel M., Poliner G., Ellis D.: Support vector machine active learning for music retrieval. ACM Multimed. Syst. J. 12, 3–13 (2006)

    Article  Google Scholar 

  • Matthews G., Jones D.M., Chamberlain A.G.: Refining the measurement of mood: the UWIST mood adjective checklist. Br. J. Psychol. 81, 17–42 (1990)

    Article  Google Scholar 

  • McFarland R.A., Kennison R.: Asymmetry in the relationship between finger temperature changes and emotional state in males. Appl. Psychophysiol. Biofeedback 14(4), 281–290 (1989)

    Google Scholar 

  • North A.C.H., David J.: Musical preferences during and after relaxation and exercise. Am. J. Psychol. 113, 43–67 (2000)

    Article  Google Scholar 

  • North A.C.H., Hargreaves D.J., Hargreaves J.J.: Uses of music in everyday life. Music Percept. 22, 41–77 (2004)

    Article  Google Scholar 

  • Oliver, N., Flores-Mangas, F.: MPTrain: a mobile, music and physiology-based personal trainer. In: Proceedings of the 8th Conference on Human–Computer Interaction with Mobile Devices and Services, pp. 21–28. ACM, New York (2006)

  • Oliver, N., Kregor-Stickles, L.: PAPA: physiology and purpose-aware automatic playlist generation. In: Lemström, K., Tindale, A., Dannenberg, R. (eds.) Proceedings of the 7th International Conference on Music Information Retrieval (ISMIR), pp. 250–253, Victoria, Canada, 8–12 October 2006

  • Ophira E., Nass C., Wagner A.D.: Cognitive control in media multitaskers. Proc. Natl Acad. Sci. 106, 15583–15587 (2009)

    Article  Google Scholar 

  • Pantic M., Patras I.: Dynamics of facial expressions: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Man Syst. Cybernet. B 36, 433–449 (2006)

    Article  Google Scholar 

  • Pelletier C.L.: The effect of music on decreasing arousal due to stress: a meta-analysis. J. Music Ther. 41, 192–214 (2004)

    Google Scholar 

  • Peter C., Herbon A.: Emotion representation and physiology assignments in digital systems. Interact. Comput. 18, 139–170 (2006)

    Article  Google Scholar 

  • Picard R.W.: Affective Computing. MIT Press, Cambridge (1997)

    Google Scholar 

  • Picard R.W.: Affective computing: challenges. Int. J. Hum.-Comput. Stud. 59, 55–64 (2003)

    Article  Google Scholar 

  • Prinz J.J.: Gut Reactions: A Perceptual Theory of Emotion. Oxford University Press, New York (2004)

    Google Scholar 

  • Rentfrow P.J., Gosling S.D.: The do re mi’s of everyday life: the structure and personality correlates of music preference. J. Pers. Soc. Psychol. 84, 1236–1256 (2003)

    Article  Google Scholar 

  • Rickard N.S.: Intense emotional responses to music: A test of the physiological arousal hypothesis. Psychol. Music 32, 371–388 (2004)

    Article  Google Scholar 

  • Rimm-Kaufman S.E., Kagan J.: The psychological significance of changes in skin temperature. Motiv. Emot. 20(1), 64–78 (1996)

    Article  Google Scholar 

  • Ritossa D.A., Rickard N.S.: The relative utility of ‘pleasantness’ and ‘liking’ dimensions in predicting the emotions expressed by music. Psychol. Music 32, 5–22 (2004)

    Article  Google Scholar 

  • Russell J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110, 145–172 (2003)

    Article  Google Scholar 

  • Rusting C.L.: Personality, mood, and cognitive processing of emotional information: three conceptual frameworks. Psychol. Bull. 124, 165–196 (1998)

    Article  Google Scholar 

  • Saarikallio S., Erkkilä J.: Role of music in adolescents’ mood regulation. Psychol. Music 35, 88–109 (2007)

    Article  Google Scholar 

  • Scott D., Sain S.: Multidimensional density estimation. In: Rao, C.R., Wegman, E.J., Solka, J.L. (eds.) Handbook of Statistics, Vol. 24, pp. 229–261. Elsevier, North Holland (2005)

    Google Scholar 

  • Silverman B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall, London (1986)

    MATH  Google Scholar 

  • Sloboda J.A.: Exploring the Musical Mind: Cognition, Emotion, Ability, Function. Oxford University Press, New York (2005)

    Google Scholar 

  • Sotiropoulos D.N., Lampropoulos A.S., Tsihrintzis G.A.: MUSIPER: a system for modeling music similarity perception based on objective feature subset selection. User Model. User-Adapt. Interact. 18, 315–348 (2008)

    Article  Google Scholar 

  • Thayer R.E.: The Biopsychology of Mood and Activation. Oxford University Press, New York (1989)

    Google Scholar 

  • Tractinsky N.: Tools over solutions? comments on interacting with computers special issue on affective computing. Interact. Comput. 16, 751–757 (2004)

    Article  Google Scholar 

  • Turlach, B.A.: Bandwidth selection in kernel density estimation: a review. Discussion Paper 9317, Institut de Statistique, Voie du Roman Pays 34, B-1348 Louvain-la-Neuve (1993)

  • Vaillant G.: Aging Well: Surprising Guideposts to a Happier Life from the Landmark Harvard Study of Adult Development. Little, Brown and Company, Boston (2003)

    Google Scholar 

  • Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M.: Guidelines for Affective Signal Processing (ASP): From lab to life. In Proceedings of the IEEE 3rd international conference on affective computing and intelligent interaction, ACII, Vol. 1, pp. 704–709. IEEE Press, Amsterdam, The Netherlands (2009a)

  • Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M., Healey, J.A.: Prerequisites for Affective Signal Processing (ASP). In: Encarnaçã, P., Veloso, A. (eds.) Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, pp. 426–433, Porto, Portugal (2009b)

  • Van den Broek E.L., Lisý V., Janssen J.H., Westerink J.H.D.M., Schut M.H., Tuinenbreijer K.: Affective man–machine interface: unveiling human emotions through biosignals. In: Fred, A., Filipe, J., Gamboa, H. (eds.) Biomedical Engineering Systems and Technologies: BIOSTEC2009 Selected Revised Papers, Vol. 52., pp. 21–47. Springer, Berlin (2010)

    Google Scholar 

  • Wagenaar W.A.: Note on the construction of digram-balanced latin squares. Psychol. Bull. 72, 384–386 (1969)

    Article  Google Scholar 

  • Webster G.D., Weir C.G.: Emotional responses to music: interactive effects of mode, texture, and tempo. Motiv. Emot. 29, 19–39 (2005)

    Article  Google Scholar 

  • Westerink, J.H.D.M., De Vries, G., Waele, S., Eerenbeemd, J., Boven, M., Ouwerkerk, M.: Emotion measurement platform for daily life situations. In: Nijholt, A., Cohn, J., Pantic, M. (eds.) Proceedings of ACII’09: Affective Computing and Intelligent Interaction, pp. 217–223. IEEE, Los Alamitos (2009)

  • Wilder J.: Stimulus and Response: The Law of Initial Values. Wright, Bristol (1967)

    Google Scholar 

  • Wilhelm P., Schoebi D.: Assessing mood in daily life: structural validity, sensitivity to change, and reliability of a short-scale to measure three basic dimensions of mood. Eur. J. Psychol. Assess. 23, 258–267 (2007)

    Article  Google Scholar 

  • Wilson G.F., Russell C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45, 635–644 (2003)

    Article  Google Scholar 

  • Yannakakis G.N., Hallam J., Lund H.H.: Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapt. Interact. 18, 207–243 (2008)

    Article  Google Scholar 

  • Zeng Z., Pantic M., Roisman G.I., Huang T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)

    Article  Google Scholar 

Download references

Acknowledgments

We gratefully acknowledge Marjolein van der Zwaag, Tim Tijs, Kathryn Segovia, and Maurits Kaptein for their helpful comments and vivid discussions on an earlier draft of this paper. We also thank three anonymous reviewers and the editor who all provided us detailed feedback on two earlier versions of this paper. Thanks to their comments and suggestions we have been able to revise this article substantially. Finally, we gratefully acknowledge Lynn Packwood for her careful proof reading.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Author information

Authors and Affiliations

  1. Human Technology Interaction, Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Den Dolech 2, 5600 MB, Eindhoven, The Netherlands

    Joris H. Janssen

  2. Department of Brain, Body, and Behavior, Philips Research, High Tech Campus 34, 5656 AE, Eindhoven, The Netherlands

    Joris H. Janssen & Joyce H. D. M. Westerink

  3. Human Media Interaction, Faculty of Electrical Engineering, Mathematics, and Computer Science, University of Twente, P.O. Box 217, 7500 AE, Enschede, The Netherlands

    Egon L. van den Broek

  4. Karakter U.C., Radboud University Medical Center Nijmegen, P.O. Box 9101, 6500 HB, Nijmegen, The Netherlands

    Egon L. van den Broek

Authors
  1. Joris H. Janssen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Egon L. van den Broek
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joyce H. D. M. Westerink
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joris H. Janssen.

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Janssen, J.H., van den Broek, E.L. & Westerink, J.H.D.M. Tune in to your emotions: a robust personalized affective music player. User Model User-Adap Inter 22, 255–279 (2012). https://doi.org/10.1007/s11257-011-9107-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-011-9107-7

Keywords

  • Mood
  • Music
  • Psychophysiology
  • User modeling
  • Kernel density estimation
  • Validation
  • Affective computing