Song Walker Harmony Space: Embodied Interaction Design for Complex Musical Skills

Part of the Springer Series on Cultural Computing book series (SSCC)


Tonal Harmony is widely considered to be the most technical and complex part of music theory. Consequently harmonic skills can be hard to acquire. Furthermore, experience of the flexible manipulation of harmony in real time generally requires the ability to play an instrument. Even for those with instrumental skills, it can be difficult to gain clear insight into harmonic abstractions. The above state of affairs gives rise to substantial barriers not only for beginners but also for many experienced musicians. To address these problems, Harmony Space is an interactive digital music system designed to give insight into a wide range of musical tasks in tonal harmony, ranging from performance and composition to analysis. Harmony Space employs a principled set of spatial mappings to offer fluid, precise, intuitive control of harmony. These mappings give rise to sensory-motor and music-theoretic affordances that are hard to obtain in any other way. As a result, harmonic abstractions are rendered amenable to concrete, visible control by simple spatial manipulation. In the language of conceptual metaphor theory, many relationships in tonal harmony become accessible to rapid, universal, low-level, robust human inference mechanisms using image schemata such as containment, contact, centre-periphery, and source-path-goal. This process is more rapid, and imposes far less cognitive load, than slow, abstract symbolic reasoning. Using the above principles, several versions of Harmony Space have been designed to exploit specific interaction styles for different purposes. We note some key variants, such as the desktop version, the camera tracked version, while focusing principally on the most recent version, Song Walker, which employs whole body interaction. Preliminary results from a study of the Song Walker system are outlined, in which both beginners and expert musicians undertook a range of musical tasks involving the performance, composition and analysis of music. Finally, we offer a discussion of the limitations of the current system, and outline directions for future work.


  1. Balzano, G. J. (1980). The group-theoretic description of 12-fold and microtonal pitch systems. Computer Music Journal, 4(4), 66–84.CrossRefGoogle Scholar
  2. Bird, J., Holland, S., Marshall, P., Rogers, Y., & Clark, A. (2008). Feel the force: Using tactile technologies to investigate the extended mind. In Proceedings of DAP 2008, Workshop on Devices that Alter Perception, 21 September 2008, Seoul, South Korea, pp. 1–4.Google Scholar
  3. Fauconnier, G., & Turner, M. (2002). How we think: Conceptual blending and the mind’s hidden complexities. New York: Basic Books. ISBN 978-0-465-08786-0.Google Scholar
  4. Holland, S. (1989). Artificial intelligence, education and music. Ph.D. thesis, Institute of Educational Technology, The Open University, Milton Keynes, UK. Published as OU IET CITE Report No. 88, July 1989.Google Scholar
  5. Holland, S. (1994). Learning about harmony with harmony space: An overview. In M. Smith & G. Wiggins (Eds.), Music education: An artificial intelligence approach (pp. 24–40). London: Springer.CrossRefGoogle Scholar
  6. Holland, S., Marshall, P., Bird, J., Dalton, S.N., Morris, R., Pantidi, N., Rogers, Y., & Clark, A. (2009). Running up Blueberry Hill: Prototyping whole body interaction in harmony space. In Proceedings of TEI 2009, the 3rd conference on tangible and embodied interaction (pp. 92–98). ISBN 978-1-60558-493-5. New York: ACM.Google Scholar
  7. Holland, S., Wilkie, K., Bouwer, A., Dalgleish, M., & Mulholland, P. (2011). Whole body interaction in abstract domains. In D. England (Ed.), Whole body interaction (Human–computer interaction series). London: Springer. ISBN 978-0-85729-432-6.Google Scholar
  8. Hurtienne, J., & Blessing, L. (2007). Design for intuitive use-testing image schema theory for user interface design. In J.-C. Bocquet (Ed.), Proceedings of ICED 07, international conference on engineering design, Paris, France, 28–31 August 2007, pp 1–12.Google Scholar
  9. Hurtienne, J., Israel, J. H., & Weber, K. (2008). Cooking up real world business applications combining physicality, digitality, and image schemas. In Proceedings of the 2nd international conference on tangible and embedded interaction (Bonn, 2008) (pp. 239–246). New York: ACM.Google Scholar
  10. Lakoff G, & Núñez, R. E. (2000). Conceptual metaphor. Where mathematics comes from: How the embodied mind brings mathematics into being (pp. 39–48). New York: Basic Books.Google Scholar
  11. Longuet-Higgins, H. C. (1962). Letter to a musical friend. Music Review, August 1962. pp. 244–248.Google Scholar
  12. O’Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences, 24(5), 939–973.CrossRefGoogle Scholar
  13. Pedler, D. (2001). The songwriting secrets of the Beatles. London: Omnibus Press. ISBN 13: 978-0711981676.Google Scholar
  14. Wilkie, K., Holland, S., & Mulholland, P. (2009). Evaluating musical software using conceptual metaphors. In A. Blackwell (Ed.), Proceedings of BCS HCI 2009, September 1–5, 2009 (pp. 232–237). Cambridge, UK: EWIC. ISSN 1477–9358. Oro ID 17967.Google Scholar
  15. Wilkie, K., Holland, S., & Mulholland, P. (2010). What can the language of musicians tell us about music interaction design? Computer Music Journal, 34(4), 34–48.CrossRefGoogle Scholar
  16. Wilkie, K., Holland, S., & Mulholland, P. (2013). Towards a participatory approach for interaction design based on conceptual metaphor theory: A case study from music interaction. In S. Holland, K. Wilkie, P. Mulholland, & A. Seago (Eds.), Music and human computer interaction (pp. –). London: Springer. ISBN 978-1-4471-2989-9.Google Scholar
  17. Zbikowski, L. M. (1997). Conceptual models and cross-domain mapping: new perspectives on theories of music and hierarchy. Journal of Music Theory, 41(2), 193–225.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  1. 1.Faculty of Science, Intelligent Systems LabUniversity of AmsterdamAmsterdamThe Netherlands
  2. 2.Music Computing Lab, Centre For Research in ComputingThe Open UniversityMilton KeynesUK
  3. 3.Department of Music, SSPALUniversity of WolverhamptonWest MidlandsUK

Personalised recommendations