The Six Emotion-Face Clock as a Tool for Continuously Rating Discrete Emotional Responses to Music

  • Emery Schubert
  • Sam Ferguson
  • Natasha Farrar
  • David Taylor
  • Gary E. McPherson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7900)

Abstract

Recent instruments measuring continuous self-reported emotion responses to music have tended to use dimensional rating scale models of emotion such as valence (happy to sad). However, numerous retrospective studies of emotion in music use checklist style responses, usually in the form of emotion words, (such as happy, angry, sad…) or facial expressions. A response interface based on six simple sketch style emotion faces aligned into a clock-like distribution was developed with the aim of allowing participants to quickly and easily rate emotions in music continuously as the music unfolded. We tested the interface using six extracts of music, one targeting each of the six faces: ‘Excited’ (at 1 o’clock), ‘Happy’ (3), ‘Calm’ (5), ‘Sad’ (7), ‘Scared’ (9) and ‘Angry’ (11). 30 participants rated the emotion expressed by these excerpts on our ‘emotion-face-clock’. By demonstrating how continuous category selections (votes) changed over time, we were able to show that (1) more than one emotion-face could be expressed by music at the same time and (2) the emotion face that best portrayed the emotion the music conveyed could change over time, and (3) the change could be attributed to changes in musical structure. Implications for research on orientation time and mixed emotions are discussed.

Keywords

Emotion in music continuous response discrete emotions time-series analysis film music 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Yang, Y.H., Lin, Y.C., Su, Y.F., Chen, H.H.: A regression approach to music emotion recognition. IEEE Transactions on Audio, Speech, and Language Processing 16(2), 448–457 (2008)CrossRefGoogle Scholar
  2. 2.
    Schmidt, E.M., Turnbull, D., Kim, Y.E.: Feature selection for content-based, time-varying musical emotion regression. In: MIR 2010 Proceedings of the International Conference on Multimedia Information Retrieval. ACM, New York (2010)Google Scholar
  3. 3.
    Korhonen, M.D., Clausi, D.A., Jernigan, M.E.: Modeling emotional content of music using system identification. IEEE Transactions on Systems Man and Cybernetics Part B-Cybernetics 36(3), 588–599 (2006)CrossRefGoogle Scholar
  4. 4.
    Schubert, E.: Continuous self-report methods. In: Juslin, P.N., Sloboda, J.A. (eds.) Handbook of Music and Emotion: Theory, Research, Applications, pp. 223–253. OUP, Oxford (2010)Google Scholar
  5. 5.
    Madsen, C.K., Frederickson, W.E.: The experience of musical tension: A replication of Nielsen’s research using the continuous response digital interface. Journal of Music Therapy 30(1), 46–63 (1993)CrossRefGoogle Scholar
  6. 6.
    Nielsen, F.V.: Musical tension and related concepts. In: Sebeok, T.A., Umiker-Sebeok, J. (eds.) The Semiotic Web 1986. An International Year-Book. Mouton de Gruyter, Berlin (1987)Google Scholar
  7. 7.
    Russell, J.A.: Affective space is bipolar. Journal of Personality and Social Psychology 37(3), 345–356 (1979)CrossRefGoogle Scholar
  8. 8.
    Russell, J.A.: A circumplex model of affect. Journal of Social Psychology 39, 1161–1178 (1980)Google Scholar
  9. 9.
    Krumhansl, C.L.: An exploratory study of musical emotions and psychophysiology. Canadian Journal of Experimental Psychology 51(4), 336–352 (1997)CrossRefGoogle Scholar
  10. 10.
    Cowie, R., Douglas-Cowie, E., Savvidou, S., McMahon, E., Sawey, M., Schröder, M.: FEELTRACE: An instrument for recording perceived emotion in real time. In: Cowie, R., Douglas-Cowie, E., Schroede, M. (eds.) Speech and Emotion: Proceedings of the ISCA workshop, pp. 19–24. Newcastle, Co. Down (2000)Google Scholar
  11. 11.
    Nagel, F., Kopiez, R., Grewe, O., Altenmüller, E.: EMuJoy: Software for continuous measurement of perceived emotions in music. Behavior Research Methods 39(2), 283–290 (2007)CrossRefGoogle Scholar
  12. 12.
    Schubert, E.: Measuring emotion continuously: Validity and reliability of the two-dimensional emotion-space. Australian Journal of Psychology 51(3), 154–165 (1999)CrossRefGoogle Scholar
  13. 13.
    Schimmack, U., Rainer, R.: Experiencing activation: Energetic arousal and tense arousal are not mixtures of valence and activation. Emotion 2(4), 412–417 (2002)CrossRefGoogle Scholar
  14. 14.
    Schimmack, U., Grob, A.: Dimensional models of core affect: A quantitative comparison by means of structural equation modeling. European Journal of Personality 14(4), 325–345 (2000)CrossRefGoogle Scholar
  15. 15.
    Wundt, W.: Grundzüge der physiologischen Psychologie. Engelmann, Leipzig (1905)Google Scholar
  16. 16.
    Plutchik, R.: The emotions: Facts, theories and a new model. Random House, New York (1962)Google Scholar
  17. 17.
    Russell, J.A., Mehrabian, A.: Evidence for a 3-factor theory of emotions. Journal of Research in Personality 11(3), 273–294 (1977)CrossRefGoogle Scholar
  18. 18.
    Barrett, L.F., Wager, T.D.: The Structure of Emotion: Evidence From Neuroimaging Studies. Current Directions in Psychological Science 15(2), 79–83 (2006)CrossRefGoogle Scholar
  19. 19.
    Barrett, L.F.: Discrete emotions or dimensions? The role of valence focus and arousal focus. Cognition & Emotion 12(4), 579–599 (1998)CrossRefGoogle Scholar
  20. 20.
    Lewis, M., Haviland, J.M. (eds.): Handbook of emotions (1993)Google Scholar
  21. 21.
    Izard, C.E.: The psychology of emotions. Plenum Press, New York (1991)CrossRefGoogle Scholar
  22. 22.
    Izard, C.E.: Organizational and motivational functions of discrete emotions. In: Lewis, M., Haviland, J.M. (eds.) Handbook of Emotions, pp. 631–641. The Guilford Press, New York (1993)Google Scholar
  23. 23.
    Namba, S., Kuwano, S., Hatoh, T., Kato, M.: Assessment of musical performance by using the method of continuous judgment by selected description. Music Perception 8(3), 251–275 (1991)CrossRefGoogle Scholar
  24. 24.
    Juslin, P.N., Laukka, P.: Communication of emotions in vocal expression and music performance: Different channels, same code? Psychological Bulletin 129(5), 770–814 (2003)CrossRefGoogle Scholar
  25. 25.
    Laukka, P., Gabrielsson, A., Juslin, P.N.: Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion. International Journal of Psychology 35(3-4), 288 (2000)Google Scholar
  26. 26.
    Juslin, P.N.: Communicating emotion in music performance: A review and a theoretical framework. In: Juslin, P.N., Sloboda, J.A. (eds.) Music and Emotion: Theory and Research, pp. 309–337. Editors. Oxford University Press, London (2001)Google Scholar
  27. 27.
    Schubert, E., Ferguson, S., Farrar, N., McPherson, G.E.: Sonification of Emotion I: Film Music. In: The 17th International Conference on Auditory Display (ICAD-2011), International Community for Auditory Display (ICAD), Budapest (2011)Google Scholar
  28. 28.
    Hevner, K.: Expression in music: a discussion of experimental studies and theories. Psychological Review 42, 187–204 (1935)CrossRefGoogle Scholar
  29. 29.
    Hevner, K.: The affective character of the major and minor modes in music. American Journal of Psychology 47, 103–118 (1935)CrossRefGoogle Scholar
  30. 30.
    Hevner, K.: Experimental studies of the elements of expression in music. American Journal of Psychology 48, 246–268 (1936)CrossRefGoogle Scholar
  31. 31.
    Hevner, K.: The affective value of pitch and tempo in music. American Journal of Psychology 49, 621–630 (1937)CrossRefGoogle Scholar
  32. 32.
    Rigg, M.G.: The mood effects of music: A comparison of data from four investigators. The Journal of Psychology 58(2), 427–438 (1964)CrossRefGoogle Scholar
  33. 33.
    Han, B., Rho, S., Dannenberg, R.B., Hwang, E.: SMERS: Music emotion recognition using support vector regression. In: Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR 2009), Kobe International Conference Center, Kobe, Japan, October 26-30 (2009)Google Scholar
  34. 34.
    Schlosberg, H.: The description of facial expressions in terms of two dimensions. Journal of Experimental Psychology 44, 229–237 (1952)CrossRefGoogle Scholar
  35. 35.
    Russell, J.A.: Reading emotion from and into faces: Resurrecting a dimensional-contextual perspective. In: Russell, J.A., Fernández-Dols, J.M. (eds.) The Psychology of Facial Expression, pp. 295–320. Cambridge University Press, Cambridge (1997)CrossRefGoogle Scholar
  36. 36.
    Dimberg, U., Thunberg, M.: Rapid facial reactions to emotional facial expressions. Scandinavian Journal of Psychology 39(1), 39–45 (1998)CrossRefGoogle Scholar
  37. 37.
    Britton, J.C., Taylor, S.F., Sudheimer, K.D., Liberzon, I.: Facial expressions and complex IAPS pictures: common and differential networks. Neuroimage 31(2), 906–919 (2006)CrossRefGoogle Scholar
  38. 38.
    Waller, B.M., Cray Jr, J.J., Burrows, A.M.: Selection for universal facial emotion. Emotion 8(3), 435–439 (2008)CrossRefGoogle Scholar
  39. 39.
    Ekman, P.: Facial expression and emotion. American Psychologist 48(4), 384–392 (1993)CrossRefGoogle Scholar
  40. 40.
    Lang, P.J.: Behavioral treatment and bio-behavioral assessment: Computer applications. In: Sidowski, J.B., Johnson, J.H., Williams, T.A. (eds.) Technology in Mental Health Care Delivery Systems, pp. 119–137. Ablex, Norwood (1980)Google Scholar
  41. 41.
    Bradley, M.M., Lang, P.J.: Measuring emotion - the self-assessment mannequin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry 25(1), 49–59 (1994)CrossRefGoogle Scholar
  42. 42.
    Ekman, P., Rosenberg, E.L. (eds.): What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Series in affective science. Oxford University Press, London (1997)Google Scholar
  43. 43.
    Eerola, T., Vuoskoski, J.K.: A comparison of the discrete and dimensional models of emotion in music. Psychology of Music 39(1), 18–49 (2011)CrossRefGoogle Scholar
  44. 44.
    Kratus, J.: A developmental study of children’s interpretation of emotion in music. Psychology of Music 21, 3–19 (1993)CrossRefGoogle Scholar
  45. 45.
    Schubert, E., McPherson, G.E.: The perception of emotion in music. In: McPherson, G.E. (ed.) The Child as Musician: A Handbook of Musical Development, pp. 193–212. Oxford University Press, Oxford (2006)CrossRefGoogle Scholar
  46. 46.
    Ekman, R., Friesen, W.V., Ellsworth, R.: Emotion in the human face: Guidelines jbr research and an integration of findings. Pergamon Press, New York (1972)Google Scholar
  47. 47.
    Zentner, M., Grandjean, D., Scherer, K.R.: Emotions evoked by the sound of music: characterization, classification, and measurement. Emotion 8(4), 494–521 (2008)CrossRefGoogle Scholar
  48. 48.
    Schubert, E.: Update of the Hevner adjective checklist. Perceptual and Motor Skills 96(3), 1117–1122 (2003)CrossRefGoogle Scholar
  49. 49.
    Kostov, V., Yanagisawa, H., Johansson, M., Fukuda, S.: Method for Face-Emotion Retrieval Using A Cartoon Emotional Expression Approach. JSME International Journal Series C 44(2), 515–526 (2001)CrossRefGoogle Scholar
  50. 50.
    Schubert, E.: Continuous measurement of self-report emotional response to music. In: Juslin, P.N., Sloboda, J.A. (eds.) Music and Emotion: Theory and Research, pp. 393–414. Oxford University Press, Oxford (2001)Google Scholar
  51. 51.
    Schubert, E.: Reliability issues regarding the beginning, middle and end of continuous emotion ratings to music. Psychology of Music 41(3), 350–371 (2013)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Bachorik, J.P., Bangert, M., Loui, P., Larke, K., Berger, J., Rowe, R., Schlaug, G.: Emotion in motion: Investigating the time-course of emotional judgments of musical stimuli. Music Perception 26(4), 355–364 (2009)CrossRefGoogle Scholar
  53. 53.
    Schubert, E., Dunsmuir, W.: Regression modelling continuous data in music psychology. In: Yi, S.W. (ed.) Music, Mind, and Science, pp. 298–352. Seoul National University, Seoul (1999)Google Scholar
  54. 54.
    Juslin, P.N., Friberg, A., Bresin, R.: Toward a computational model of expression in music performance: The GERM model. Musicae Scientiae. Special Issue: pp. 63-122 (2001)Google Scholar
  55. 55.
    Lucas, B.J., Schubert, E., Halpern, A.R.: Perception of emotion in sounded and imagined music. Music Perception 27(5), 399–412 (2010)CrossRefGoogle Scholar
  56. 56.
    Upham, F.: Quantifying the temporal dynamics of music listening: A critical investigation of analysis techniques for collections of continuous responses to music. McGill University (2011)Google Scholar
  57. 57.
    Schubert, E., Vincs, K., Stevens, C.J.: Identifying regions of good agreement among responders in engagement with a piece of live dance. Empirical Studies of the Arts 13(1), 1–20 (2013)CrossRefGoogle Scholar
  58. 58.
    Hunter, P.G., Schellenberg, E.G., Schimmack, U.: Mixed affective responses to music with conflicting cues. Cognition & Emotion 22(2), 327–352 (2008)CrossRefGoogle Scholar
  59. 59.
    Hunter, P.G., Schellenberg, E.G., Schimmack, U.: Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts 4(1), 47–56 (2010)CrossRefGoogle Scholar
  60. 60.
    Trohidis, K., Tsoumakas, G., Kalliris, G., Vlahavas, I.: Multilabel classification of music into emotions. In: Proceedings of the 9th International Conference on Music Information Retrieval (ISMIR 2008), Philadelphia, PA (2008)Google Scholar
  61. 61.
    Levy, M., Sandler, M.: A semantic space for music derived from social tags. In: Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR 2007), Vienna, Austria (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Emery Schubert
    • 1
  • Sam Ferguson
    • 2
  • Natasha Farrar
    • 1
  • David Taylor
    • 1
  • Gary E. McPherson
    • 3
  1. 1.Empirical Musicology GroupUniversity of New South WalesSydneyAustralia
  2. 2.University of TechnologySydneyAustralia
  3. 3.Melbourne Conservatorium of MusicUniversity of MelbourneMelbourneAustralia

Personalised recommendations