Human Nonverbal Behaviour Understanding in the Wild for New Media Art

  • Evan Morgan
  • Hatice Gunes
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8212)


Over the course of the London 2012 Olympics a large public installation took place in Central London. Its premise was to enable members of the public to express themselves by controlling the lights around the rim of the London Eye. The installation’s design and development was undertaken as a collaborative project between an interactive arts studio and researchers in the field of affective and behavioural computing. Over 800 people participated, taking control of the lights using their heart rates and hand gestures. This paper approaches nonverbal and affective behaviour understanding for new media art as a case study, and reports the design of this installation and the subsequent analysis of over one million frames of physiological and motion capture data. In doing so it sheds light on how the intersection of affective and behavioural computing and new media art could be beneficial to both researchers and artists.


New media art affective behaviour understanding in the wild gestural interaction mood nonverbal behaviour 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Tribe, M., Jana, R.: New Media Art. Taschen (2006)Google Scholar
  2. 2.
    Isen, A.M., Daubman, K.A., Nowicki, G.P.: Positive affect facilitates creative problem solving. J. of Personality & Social Psychology 52(6), 1122–1131 (1987)CrossRefGoogle Scholar
  3. 3.
    Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12), 2067–2083 (2008)CrossRefGoogle Scholar
  4. 4.
    Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Medical & Biological Engineering & Computing 42(3), 419–427 (2004)CrossRefGoogle Scholar
  5. 5.
    Picard, R., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
  6. 6.
    Allanson, J., Fairclough, S.H.: A research agenda for physiological computing. Interacting with Computers 16(5), 857–878 (2004)CrossRefGoogle Scholar
  7. 7.
    Ravaja, N., Saari, T., Salminen, M., Laarni, J., Kallinen, K.: Phasic emotional reactions to video game events: A psychophysiological investigation. Media Psychology 8(4), 343–367 (2006)CrossRefGoogle Scholar
  8. 8.
    Knapp, R.B., Jaimovich, J., Coghlan, N.: Measurement of motion and emotion during musical performance. In: 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pp. 1–5. IEEE (September 2009)Google Scholar
  9. 9.
    Anttonen, J., Surakka, V.: Emotions and heart rate while sitting on a chair. In: Proc. CHI, p. 491 (2005)Google Scholar
  10. 10.
    Billinghurst, M.: Gesture based interaction. In: Haptic Input, pp. 1–35 (2011)Google Scholar
  11. 11.
    Sturman, D.J., Zeltzer, D.: A design method for whole-hand human-computer interaction. ACM Tran. on Information Systems 11(3), 219–238 (1993)CrossRefGoogle Scholar
  12. 12.
    Castro, B.P., Velho, L., Kosminsky, D.: Integrarte: digital art using body interaction. In: Proc. of the Annual Symposium on Computational Aesthetics in Graphics, Visualization, and Imaging, pp. 11–15 (2012)Google Scholar
  13. 13.
    Visnjic, F., Smith, G.J., Scholz, A.: Creative applications (2013)Google Scholar
  14. 14.
    Scherer, K.R.: Emotion as a Multicomponent Process: A model and some cross-cultural data. Review of Personality and Social Psychology 5, 37–63 (1984)Google Scholar
  15. 15.
    Fox, E.: Emotion Science Cognitive and Neuroscientific Approaches to Understanding Human Emotions. Palgrave Macmillan (September 2008)Google Scholar
  16. 16.
    Pollick, F.E., Paterson, H.M., Bruderlin, A., Sanford, A.J.: Perceiving affect from arm movement. Cognition 82(2), B51–B61 (2001)Google Scholar
  17. 17.
    Russell, J.A.: A circumplex model of affect. J. of Personality & Social Psychology 39(6), 1161–1178 (1980)CrossRefGoogle Scholar
  18. 18.
    Castellano, G., Villalba, S.D., Camurri, A.: Recognising human emotions from body movement and gesture dynamics. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 71–82. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  19. 19.
    Saerbeck, M., Bartneck, C.: Perception of affect elicited by robot motion. In: Proc. of ACM/IEEE Int’l Conf. on Human-Robot Interaction, pp. 53–60 (2010)Google Scholar
  20. 20.
    Wei, S.X.: Resistance is fertile: Gesture and agency in the field of responsive media. Configurations 10(3), 439–472 (2002)CrossRefGoogle Scholar
  21. 21.
    Haken, H., Kelso, J.A.S., Bunz, H.: A theoretical model of phase transitions in human hand movements. Biological Cybernetics 51(5), 347–356 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Alaoui, S.F., Caramiaux, B., Serrano, M., Bevilacqua, F.: Movement qualities as interaction modality. In: Proceedings of the Designing Interactive Systems Conference on DIS 2012, pp. 761–769. ACM Press, New York (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Evan Morgan
    • 1
  • Hatice Gunes
    • 1
  1. 1.School of Electronic Engineering and Computer ScienceQueen Mary University of LondonLondonU.K

Personalised recommendations