Psychophysiological Measures of Emotional Response to Romantic Orchestral Music and Their Musical and Acoustic Correlates

  • Konstantinos Trochidis
  • David Sears
  • Diêu-Ly Trân
  • Stephen McAdams
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7900)


This paper examines the induction of emotions while listening to Romantic orchestral music. The study seeks to explore the relationship between subjective ratings of felt emotion and acoustic and physiological features. We employed 75 musical excerpts as stimuli to gather responses of excitement and pleasantness from 20 participants. During the experiments, physiological responses of the participants were measured, including blood volume pulse (BVP), skin conductance (SC), respiration rate (RR) and facial electromyography (EMG). A set of acoustic features was derived related to dynamics, harmony, timbre and rhythmic properties of the music stimuli. Based on the measured physiological signals, a set of physiological features was also extracted. The feature extraction process is discussed with particular emphasis on the interaction between acoustical and physiological parameters. Statistical relations among audio, physiological features and emotional ratings from psychological experiments were systematically investigated. Finally, a forward step-wise multiple linear regression model (MLR) was employed using the best features, and its prediction efficiency was evaluated and discussed. The results indicate that merging acoustic and physiological modalities substantially improves prediction of participants’ ratings of felt emotion compared to the results using the modalities in isolation.


Heart Rate Variability Emotion Recognition Skin Conductance Acoustic Feature Audio Feature 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Gabrielsson, A., Lindström, E.: The role of structure in the musical expression of emotions. In: Juslin, P.N., Sloboda, J.A. (eds.) Music and Emotion: Theory, Research, Applications. Oxford University Press, Oxford (2010)Google Scholar
  2. 2.
    Gomez, P., Danuser, B.: Relationships between musical structure and physiological measures of emotion. Emotion 7(2), 377–387 (2004)CrossRefGoogle Scholar
  3. 3.
    Li, T., Ogihara, M.: Detecting emotion in music. In: Proceedings of the International Conference for Music Information Retrieval (ISMIR), Baltimore (2003)Google Scholar
  4. 4.
    Lu, L., Lu, D., Zhang, H.J.: Automatic mood detection and tracking of music audio signal. IEEE Transactions on Audio, Speech and Language Processing 14(1), 5–18 (2006)CrossRefGoogle Scholar
  5. 5.
    Tzanetakis, G.: Marsyas submission to MIREX 2007. MIREX (2007)Google Scholar
  6. 6.
    Peeters, G.: A generic training and classification system for MIREX08 classification tasks: Audio music mood, audio genre, audio artist, audio tag. MIREX (2008)Google Scholar
  7. 7.
    Kim, Y.E., Schmidt, E.M., Migneco, R., Morton, B.G., Richardson, P., Scott, J., Speck, J.A., Turnbull, D.: Music emotion recognition: a state of the art review. In: Proceedings of the International Conference for Music Information Retrieval (ISMIR), pp. 255–266 (2010)Google Scholar
  8. 8.
    Trochidis, K., Tsoumakas, G., Kalliris, G., Vlahavas, I.: Multi-label classification of music into emotions. In: Proceedings of the International Conference for Music Information Retrieval, ISMIR (2008)Google Scholar
  9. 9.
    Song, Y., Simon, D., Pears, M.: Evaluation of musical features for emotion classification. In: Proceedings of the International Conference for Music Information Retrieval (ISMIR), pp. 523–528 (2012)Google Scholar
  10. 10.
    Schmidt, E.M., Turnbull, D., Kim, Y.E.: Feature selection for content-based, time-varying musical emotion regression. In: Proceedings of the International Conference for Music Information Retrieval (ISMIR), pp. 267–274 (2010)Google Scholar
  11. 11.
    Mc Dornan, K.F., Ough, S., Ho, C.C.: Automatic emotion prediction of song excerpts: Index construction, algorithm design and empirical comparison. Journal of New Music Research 36(4), 281–299 (2007)CrossRefGoogle Scholar
  12. 12.
    Eerola, T.: Are the Emotions Expressed in Music Genre-specific? An Audio-based Evaluation of Datasets Spanning Classical, Film, Pop and Mixed Genres. Journal of New Music Research 40(4), 349–366 (2011)CrossRefGoogle Scholar
  13. 13.
    Yang, Y.-H., Lin, Y.-C., Cheng, H.-T., Liao, I.-B., Ho, Y.-C., Chen, H.H.: Toward multi-modal music emotion classification. In: Huang, Y.-M.R., Xu, C., Cheng, K.-S., Yang, J.-F.K., Swamy, M.N.S., Li, S., Ding, J.-W. (eds.) PCM 2008. LNCS, vol. 5353, pp. 70–79. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Laurier, C., Sordo, M., Serra, J., Herrera, P.: Music mood representation from social tags. In: Proceedings of the International Conference for Music Information Retrieval, ISMIR (2009)Google Scholar
  15. 15.
    Hu, Y., Chen, X., Yang, D.: Lyrics based song emotion detection with affective lexicon and fuzzy clustering method. In: Proceedings of the International Conference for Music Information Retrieval, ISMIR (2009)Google Scholar
  16. 16.
    Schuller, B., Dorfner, J., Rigoll, D.: Determination of non-prototypical valence and arousal in popular music: features and performances. EURASIP Journal on Audio, Speech and Music Processing 2010, 1–20 (2010)Google Scholar
  17. 17.
    Turnbull, D., Barrington, L., Torres, D., Lanckiert, G.: Semantic annotation and retrieval of music and sound effects. IEEE Transactions on Audio, Speech and Language Processing 16(2), 455–462 (2010)Google Scholar
  18. 18.
    Biscoff, K., Firan, C.S., Paiu, R., Nejdl, W., Laurier, C., Sodo, M.: Proceedings of the International Conference for Music Information Retrieval, ISMIR (2009)Google Scholar
  19. 19.
    Dunker, P., Nowak, S., Begau, N., Lanz, C.: Content-based mood classification framework and evaluation approach. In: Proceedings of ACM, New York (2008)Google Scholar
  20. 20.
    Nyklicek, I., Thayer, J., Van Doornen, L.: Cardiorespiratory differentiation of musically-induced emotion. Journal of Psychophysiology 11, 304–321 (1997)Google Scholar
  21. 21.
    Krumhansl, C.: An explanatory study of musical emotion and psychophysiology. Canadian Journal of Experimental Psychology 51, 336–352 (1997)CrossRefGoogle Scholar
  22. 22.
    Khalfa, S., Peretz, I., Blondin, J., Manon, R.: Event-related skin conductance responses to music al emotion in humans. Neuroscience Letters 328, 145–149 (2002)CrossRefGoogle Scholar
  23. 23.
    Lundquist, L., Carlsson, F., Hilmersson, P.: Facial electromyography, autonomic activity and emotional experience to happy and sad music. Paper Presented at the International Congress of Psychology (2002)Google Scholar
  24. 24.
    Nasoz, F., Lisetti, C.L., Alvarez, K., Finkelstein, N.: Emotional Recognition from Physiological Signals for User Modeling of Affect. In: Proceedings of the 3rd Workshop on Affective and Attitude User Modeling (2003)Google Scholar
  25. 25.
    Wagner, J., Kim, J., Andre, E.: From physiological signals to emotion. In: International Conference on Multimedia and Expo, pp. 940–943 (2005)Google Scholar
  26. 26.
    Kim, J.: Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12), 2067–2083 (2008)CrossRefGoogle Scholar
  27. 27.
    Koelstra, S., Muehl, C., Soleymani, M., Lee, J.D., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: DEAP: A database for emotion analysis using physiological signals. IEEE Transactions on Affective Computing 3(1), 18–31 (2011)CrossRefGoogle Scholar
  28. 28.
    Ekman, P.: Are there basic emotions? Physiological Review 99(3), 550–553 (1992)Google Scholar
  29. 29.
    Russel, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology 39(6), 1161–1178 (1980)CrossRefGoogle Scholar
  30. 30.
    Withvliet, C.V., Vrana, S.R.: Play it again Sam: repeated exposure to emotionally evocative music polarizes liking and smiling responses and influences the affective reports, facial EMG and heart rate. Cognition & Emotion 21(1), 1–23 (2006)Google Scholar
  31. 31.
    Eerola, T., Vuoskoski, J.K.: A comparison of discrete and dimensional models of emotion in music. Psychology of Music 31(1), 18–49 (2010)Google Scholar
  32. 32.
    Bradley, M.M., Lang, P.J.: Emotion and Motivation. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, 3rd edn., pp. 581–607. Cambridge University Press, New York (2008)Google Scholar
  33. 33.
    Kim, J., Andre, E., Rehm, M., Vogt, T., Wagner, J.: Integrating information from speech and physiological signals to achieve emotion sensitivity. INTERSPEECH 2005, 809–812 (2005)Google Scholar
  34. 34.
    Kim, J., Andre, E.: Emotion recognition using physiological and speech signals in short term observation. In: ICGI 2006. LNCS (LNAI), vol. 4201, pp. 53–64. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  35. 35.
    Dibben, N.: The role of peripheral feedback in emotional experience with music. Music Perception 22(1), 79–115 (2002)CrossRefGoogle Scholar
  36. 36.
    Scherer, K., Zentner, M.: Emotional effects of music: Production rules. In: Juslin, P., Sloboda, J. (eds.) Music and Emotion: Theory and Research, Oxford University Press, Oxford (2001)Google Scholar
  37. 37.
    Bigand, E., Vieillard, S., Madurell, F., Marozeau, J., Dacquet, A.: Multidimensional scaling of emotional responses to music: The effect of musical expertise and of the duration of the excerpts. Cognition & Emotion 19(8), 1113–1139 (2005)CrossRefGoogle Scholar
  38. 38.
    Dimberg, U.: Facial electromyography and emotional reactions. Psychophysiology 27(5), 481–494 (1990)CrossRefGoogle Scholar
  39. 39.
    Rickard, N.: Intense emotional responses to music: a test of the physiological arousal hypothesis. Psychology of Music 32(4), 371–399 (2004)CrossRefGoogle Scholar
  40. 40.
    Lartillot, O., Toiviainen, P.: MIR in Matlab (II): A Toolbox for Musical Feature Extraction From Audio. In: International Conference on Music Information Retrieval, Vienna (2007)Google Scholar
  41. 41.
    Sethares, W.: Tuning, Timbre, Spectrum, Scale. Springer, Berlin (1998)CrossRefGoogle Scholar
  42. 42.
    Gomez, E.: Tonal description of polyphonic audio for music content processing. INFORMS Journal on Computing 18(3), 294–304 (2006)CrossRefGoogle Scholar
  43. 43.
    Harte, C., Sandler, M., Gasser, M.: Detecting harmonic change in musical audio. In: Proceedings of the 1st ACM Workshop on Audio and Music Computing Multimedia, Santa Barbara, CA, pp. 26–31 (2006)Google Scholar
  44. 44.
    Saari, P., Eerola, T., Lartillot, O.: Generalizability and simplicity as criteria in feature selection: Application to mood classification in music. IEEE Transactions in Audio, Language, and Speech Processing 19(6), 1802–1812 (2011)CrossRefGoogle Scholar
  45. 45.
    Tolonen, T., Karjalainen, M.: A computationally efficient multipitch analysis model. IEEE Transactions on Speech and Audio Processing 8(6), 708–716 (2000)CrossRefGoogle Scholar
  46. 46.
    Pampalk, E., Rauber, A., Merkl, D.: Content based organization and visualization of music archives. In: Proceedings of the 10th ACM International Conference on Multimedia, Juan les Pins, France, pp. 579–585 (2002)Google Scholar
  47. 47.
    Foote, J., Cooper, M.: Media segmentation using self-similarity decomposition. In: Proceedings of SPIE Storage and Retrieval for Multimedia Databases, vol. 5021, pp. 167–175 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Konstantinos Trochidis
    • 1
  • David Sears
    • 1
  • Diêu-Ly Trân
    • 1
  • Stephen McAdams
    • 1
  1. 1.Schulich School of MusicMGill UniversityCanada

Personalised recommendations