Abstract
The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application.
Article PDF
References
Baas M., De Dreu C.K.W., Nijstad B.A.: A meta-analysis of 25 years of moodcreativity research: hedonic tone, activation, or regulatory focus?. Psychol. Bull. 134, 779–806 (2008)
Bak A.A., Grobbee D.E.: A randomized study on coffee and blood pressure. J. Hum. Hypertens. 4, 259–264 (1990)
Baumgartner T., Esslen M., Jäncke L.: From emotion perception to emotion experience: emotions evoked by pictures and classical music. Int. J. Psychophysiol. 60(1), 34–43 (2006)
Beedie C.J., Terry P.C., Lane A.M.: Distinctions between emotion and mood. Cogn. Emot. 19, 847–878 (2005)
Bishop C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)
Boehner K., DePaula R., Dourish P., Sengers P.: How emotion is made and measured. Int. J. Hum.-Comput. Stud. 65, 275–291 (2007)
Boucsein W.: Electrodermal Activity. Plenum Press, New York (1992)
Caberletti, L., Elfmann, K., Kümmel, M., Schierz, C.: Influence of ambient lighting in vehicle interior on the driver’s perception. In: de Kort, Y., IJsselsteijn, W., Vogels, I., Aarts, M., Tenner, A., Smolders, K. (eds.) Proceedings of Experiencing Light 2009 International Conference on the Effects of Light on Wellbeing, pp. 5–13, Eindhoven, The Netherlands (2009)
Cacioppo J., Tassinary L.: Inferring psychological significance from physiological signals. Am. Psychol. 45, 16–28 (1990)
Calvo R.A., D’Mello S.: Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1, 18–37 (2010)
Carberry S., de Rosis F.: Introduction to special issue on affective modeling and adaptation. User Model. User-Adapt. Interact. 18, 1–9 (2008)
Chin N.D.: Emperical evaluation of user models and user adaptive systems. User Model. User-Adapt. Interact. 11, 181–194 (2001)
Clore G.L., Palmer J.: Affective guidance of intelligent agents: how emotion controls cognition. Cogn. Syst. Res. 10(1), 21–30 (2009)
Csíkszentmihályi M.: Flow: The Psychology of Optimal Experience. Harper Collins, Sussex, UK (1990)
de Rosis F.: Preface: towards adaptation of interaction to affective factors. User Model. User-Adapt. Interact. 11, 267–278 (2001)
D’Mello S., Craig S.D., Witherspoon A., McDaniel B., Graesser A.: Automatic detection of learners affect from conversational cues. User Model. User-Adapt. Interact. 18, 45–80 (2008)
Fairclough S.H.: Fundamentals of physiological computing. Interact. Comput. 21, 133–145 (2009)
Fogg B.J.: Persuas. Technology. Morgan Kaufmann Publishers, New York (2003)
Frijda N.H.: The Emotions. Cambridge University Press, New York (1986)
Geenen R., van de Vijver F.J.R.: A simple test of the law of initial values. Psychophysiology 30(5), 525–530 (1993)
Gendolla G.H.E.: On the impact of mood on behavior: an integrative theory and a review. Rev. Gen. Psychol. 4, 378–408 (2000)
Gendolla G.H.E., Brinkman K.: The role of mood states in self-regulation: effects on action preferences and resource mobilization. Eur. Psychol. 10, 187–198 (2005)
Gendolla G.H.E., Krüsken J.: Mood state and cardiovascular response in active coping with an affect-regulative challenge. Int. J. Psychophysiol. 41, 169–180 (2001)
Hanson M.A., Powell H.C. Jr, Barth A.T., Ringgenberg K., Calhoun B.H., Aylor J.H. et al.: Body area sensor networks: challenges and opportunities. IEEE Comput. 42, 58–65 (2009)
Härdle W.: Smoothing Techniques, with Implementations in S. Springer, New York (1991)
Healey, J.A.: Affect detection in the real world: recording and processing physiological signals. In: Proceedings of the IEEE 3rd International Conference on Affective Computing and Intelligent Interaction, ACII, Vol. 1, pp. 729–734. IEEE Press, Amsterdam (2009)
Healey J.A., Picard R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6, 156–166 (2005)
Healey, J.A., Picard, R.W., Dabek, F.: A new affect-perceiving interface and its application to personalized music selection. In: Turk, M. (ed.) Proceedings of the 1998 Workshop on Perceptual User Interfaces (PUI), San Francisco, CA, USA (1998)
Heinz C., Seeger B.: Cluster kernels: resource-aware kernel density estimators over streaming data. IEEE Trans. Knowl. Data Eng. 20, 880–893 (2008)
Höök K.: Affective loop experiences: designing for interactional embodiment. Philos. Trans. R. Soc. B 364, 3585–3595 (2009)
Husain G., Thompson W.F., Schellenberg E.G.: Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Percept. 20, 151–171 (2002)
Janssen J.H., Bailenson J.N., IJsselstein W.A., Westerink J.H.D.M.: Intimate heartbeats: opportunities for affective communication technology. IEEE Trans. Affect. Comput. 1(2), 72–80 (2010)
Kaptein M., Eckles D.: Selecting effective means to any end: futures and ethics of per30 suasion profiling. In: Ploug, T., Hasle, P., Oinas-Kukkonen, H. (eds.) Persuasive Technology, pp. 82–93. Springer, Berlin (2010)
Kim J., André E.: Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2067–2083 (2008)
Kowal J., Fortier M.: Motivational determinants of flow: contributions from selfdetermination theory. J. Soc. Psychol. 139, 355–368 (1999)
Lemmens, P., Crompvoets, F., Brokken, D., van den Eerenbeemd, J., De Vries, G.-J.: A body-conforming tactile jacket to enrich movie viewing. In World Haptics Conference, pp. 7–12. IEEE, Los Alamitos (2009)
Lesiuk T.: The effect of music listening on work performance. Psychol. Music 33, 173–191 (2005)
Liljedahl, M., Sjömark, C., Lefford, N.: Using music to promote physical well-being via computer-mediated interaction. In MusicNetwork Open Workshop, 5 (2005)
Mandel M., Poliner G., Ellis D.: Support vector machine active learning for music retrieval. ACM Multimed. Syst. J. 12, 3–13 (2006)
Matthews G., Jones D.M., Chamberlain A.G.: Refining the measurement of mood: the UWIST mood adjective checklist. Br. J. Psychol. 81, 17–42 (1990)
McFarland R.A., Kennison R.: Asymmetry in the relationship between finger temperature changes and emotional state in males. Appl. Psychophysiol. Biofeedback 14(4), 281–290 (1989)
North A.C.H., David J.: Musical preferences during and after relaxation and exercise. Am. J. Psychol. 113, 43–67 (2000)
North A.C.H., Hargreaves D.J., Hargreaves J.J.: Uses of music in everyday life. Music Percept. 22, 41–77 (2004)
Oliver, N., Flores-Mangas, F.: MPTrain: a mobile, music and physiology-based personal trainer. In: Proceedings of the 8th Conference on Human–Computer Interaction with Mobile Devices and Services, pp. 21–28. ACM, New York (2006)
Oliver, N., Kregor-Stickles, L.: PAPA: physiology and purpose-aware automatic playlist generation. In: Lemström, K., Tindale, A., Dannenberg, R. (eds.) Proceedings of the 7th International Conference on Music Information Retrieval (ISMIR), pp. 250–253, Victoria, Canada, 8–12 October 2006
Ophira E., Nass C., Wagner A.D.: Cognitive control in media multitaskers. Proc. Natl Acad. Sci. 106, 15583–15587 (2009)
Pantic M., Patras I.: Dynamics of facial expressions: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Man Syst. Cybernet. B 36, 433–449 (2006)
Pelletier C.L.: The effect of music on decreasing arousal due to stress: a meta-analysis. J. Music Ther. 41, 192–214 (2004)
Peter C., Herbon A.: Emotion representation and physiology assignments in digital systems. Interact. Comput. 18, 139–170 (2006)
Picard R.W.: Affective Computing. MIT Press, Cambridge (1997)
Picard R.W.: Affective computing: challenges. Int. J. Hum.-Comput. Stud. 59, 55–64 (2003)
Prinz J.J.: Gut Reactions: A Perceptual Theory of Emotion. Oxford University Press, New York (2004)
Rentfrow P.J., Gosling S.D.: The do re mi’s of everyday life: the structure and personality correlates of music preference. J. Pers. Soc. Psychol. 84, 1236–1256 (2003)
Rickard N.S.: Intense emotional responses to music: A test of the physiological arousal hypothesis. Psychol. Music 32, 371–388 (2004)
Rimm-Kaufman S.E., Kagan J.: The psychological significance of changes in skin temperature. Motiv. Emot. 20(1), 64–78 (1996)
Ritossa D.A., Rickard N.S.: The relative utility of ‘pleasantness’ and ‘liking’ dimensions in predicting the emotions expressed by music. Psychol. Music 32, 5–22 (2004)
Russell J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110, 145–172 (2003)
Rusting C.L.: Personality, mood, and cognitive processing of emotional information: three conceptual frameworks. Psychol. Bull. 124, 165–196 (1998)
Saarikallio S., Erkkilä J.: Role of music in adolescents’ mood regulation. Psychol. Music 35, 88–109 (2007)
Scott D., Sain S.: Multidimensional density estimation. In: Rao, C.R., Wegman, E.J., Solka, J.L. (eds.) Handbook of Statistics, Vol. 24, pp. 229–261. Elsevier, North Holland (2005)
Silverman B.W.: Density Estimation for Statistics and Data Analysis. Chapman and Hall, London (1986)
Sloboda J.A.: Exploring the Musical Mind: Cognition, Emotion, Ability, Function. Oxford University Press, New York (2005)
Sotiropoulos D.N., Lampropoulos A.S., Tsihrintzis G.A.: MUSIPER: a system for modeling music similarity perception based on objective feature subset selection. User Model. User-Adapt. Interact. 18, 315–348 (2008)
Thayer R.E.: The Biopsychology of Mood and Activation. Oxford University Press, New York (1989)
Tractinsky N.: Tools over solutions? comments on interacting with computers special issue on affective computing. Interact. Comput. 16, 751–757 (2004)
Turlach, B.A.: Bandwidth selection in kernel density estimation: a review. Discussion Paper 9317, Institut de Statistique, Voie du Roman Pays 34, B-1348 Louvain-la-Neuve (1993)
Vaillant G.: Aging Well: Surprising Guideposts to a Happier Life from the Landmark Harvard Study of Adult Development. Little, Brown and Company, Boston (2003)
Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M.: Guidelines for Affective Signal Processing (ASP): From lab to life. In Proceedings of the IEEE 3rd international conference on affective computing and intelligent interaction, ACII, Vol. 1, pp. 704–709. IEEE Press, Amsterdam, The Netherlands (2009a)
Van den Broek, E.L., Janssen, J.H., Westerink, J.H.D.M., Healey, J.A.: Prerequisites for Affective Signal Processing (ASP). In: Encarnaçã, P., Veloso, A. (eds.) Biosignals 2009: Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing, pp. 426–433, Porto, Portugal (2009b)
Van den Broek E.L., Lisý V., Janssen J.H., Westerink J.H.D.M., Schut M.H., Tuinenbreijer K.: Affective man–machine interface: unveiling human emotions through biosignals. In: Fred, A., Filipe, J., Gamboa, H. (eds.) Biomedical Engineering Systems and Technologies: BIOSTEC2009 Selected Revised Papers, Vol. 52., pp. 21–47. Springer, Berlin (2010)
Wagenaar W.A.: Note on the construction of digram-balanced latin squares. Psychol. Bull. 72, 384–386 (1969)
Webster G.D., Weir C.G.: Emotional responses to music: interactive effects of mode, texture, and tempo. Motiv. Emot. 29, 19–39 (2005)
Westerink, J.H.D.M., De Vries, G., Waele, S., Eerenbeemd, J., Boven, M., Ouwerkerk, M.: Emotion measurement platform for daily life situations. In: Nijholt, A., Cohn, J., Pantic, M. (eds.) Proceedings of ACII’09: Affective Computing and Intelligent Interaction, pp. 217–223. IEEE, Los Alamitos (2009)
Wilder J.: Stimulus and Response: The Law of Initial Values. Wright, Bristol (1967)
Wilhelm P., Schoebi D.: Assessing mood in daily life: structural validity, sensitivity to change, and reliability of a short-scale to measure three basic dimensions of mood. Eur. J. Psychol. Assess. 23, 258–267 (2007)
Wilson G.F., Russell C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45, 635–644 (2003)
Yannakakis G.N., Hallam J., Lund H.H.: Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapt. Interact. 18, 207–243 (2008)
Zeng Z., Pantic M., Roisman G.I., Huang T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)
Acknowledgments
We gratefully acknowledge Marjolein van der Zwaag, Tim Tijs, Kathryn Segovia, and Maurits Kaptein for their helpful comments and vivid discussions on an earlier draft of this paper. We also thank three anonymous reviewers and the editor who all provided us detailed feedback on two earlier versions of this paper. Thanks to their comments and suggestions we have been able to revise this article substantially. Finally, we gratefully acknowledge Lynn Packwood for her careful proof reading.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Janssen, J.H., van den Broek, E.L. & Westerink, J.H.D.M. Tune in to your emotions: a robust personalized affective music player. User Model User-Adap Inter 22, 255–279 (2012). https://doi.org/10.1007/s11257-011-9107-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11257-011-9107-7