A Conceptual Framework for Music-Based Interaction Systems

  • Pieter-Jan Maes
  • Luc Nijs
  • Marc Leman
Part of the Springer Handbooks book series (SHB)


Music affords a wide range of interactive behaviors involving social, cognitive, emotional, and motor skills. In this chapter, we consider the role of technologies in relation to these interactions afforded by music. A general conceptual model is introduced that forms a basis to frame and understand a vast number of music-based interactive systems. In this model, we consider the necessity of coupled action–perception processes, in combination with human reward, prediction and social interaction processes. In addition, we discuss three perspectives on how music-based interaction systems may involve users' actions (monitoring, motivation, and alteration). To conclude, we discuss two case studies of technologies to illustrate the most innovative aspects of the presented model.


human–computer interaction


human mirror neuron


music paint machine


  1. 37.1
    J. Phillips-Silver, P.E. Keller: Searching for roots of entrainment and joint action in early musical interactions, Front. Hum. Neurosci. 6, 26 (2012)CrossRefGoogle Scholar
  2. 37.2
    A. D’Ausilio, G. Novembre, L. Fadiga, P.E. Keller: What can music tell us about social interaction?, Trends Cogn. Sci. 19(3), 111–114 (2015)CrossRefGoogle Scholar
  3. 37.3
    J. Krueger: Affordances and the musically extended mind, Front. Psychol. 4, 1003 (2013)Google Scholar
  4. 37.4
    C.M. Conway, D.B. Pisoni, W.G. Kronenberger: The importance of sound for cognitive sequencing abilities the auditory scaffolding hypothesis, Curr. Dir. Psychol. Sci. 18(5), 275–279 (2009)CrossRefGoogle Scholar
  5. 37.5
    B. Tillmann: Music and language perception: Expectations, structural integration, and cognitive sequencing, Topics Cogn. Sci. 4(4), 568–584 (2012)MathSciNetCrossRefGoogle Scholar
  6. 37.6
    R.J. Bood, M. Nijssen, J. van der Kamp, M. Roerdink: The power of auditory-motor synchronization in sports: Enhancing running performance by coupling cadence with the right beats, PLoS ONE 8(8), e70758 (2013)CrossRefGoogle Scholar
  7. 37.7
    C.I. Karageorghis, D.-L. Priest: Music in the exercise domain: A review and synthesis (Part I), Int. Rev. Sport Exerc. Psychol. 5(1), 44–66 (2012)CrossRefGoogle Scholar
  8. 37.8
    A. Moussard, E. Bigand, S. Belleville, I. Peretz: Music as a mnemonic to learn gesture sequences in normal aging and Alzheimer’s disease, Front. Hum. Neurosci. 8, 294 (2014)CrossRefGoogle Scholar
  9. 37.9
    L. Nijs, M. Lesaffre, M. Leman: The musical instrument as a natural extension of the musician. In: Music and Its Instruments, ed. by M. Castellengo, H. Genevois (Editions Delatour France, Sampzon 2013)Google Scholar
  10. 37.10
    W.I. Bauer: Music Learning Today: Digital Pedagogy for Creating, Performing, and Responding to Music (Oxford Univ. Press, Oxford 2014)CrossRefGoogle Scholar
  11. 37.11
    A. Brown: Music Technology and Education: Amplifying Musicality (Routledge, New York 2014)Google Scholar
  12. 37.12
    J. Dorfman: Theory and Practice of Technology-Based Music Instruction (Oxford Univ. Press, Oxford 2013)Google Scholar
  13. 37.13
    K.O. Hara, B. Brown: Consuming Music Together: Social and Collaborative Aspects of Music Consumption Technologies (Springer, Berlin, Heidelberg 2006)Google Scholar
  14. 37.14
    M. Leman: Embodied Music Cognition and Mediation Technology (MIT Press, Cambridge 2007)Google Scholar
  15. 37.15
    J.A. Fodor: The Modularity of Mind: An Essay on Faculty Psychology (MIT Press, Cambridge 1983)Google Scholar
  16. 37.16
    A.K. Engel, A. Maye, M. Kurthen, P. König: Where’s the action? The pragmatic turn in cognitive science, Trends Cogn. Sci. 17(5), 202–209 (2013)CrossRefGoogle Scholar
  17. 37.17
    A. Clark, D. Chalmers: The extended mind, Analysis 58(1), 7–19 (1998)CrossRefGoogle Scholar
  18. 37.18
    C. Addyman, R.M. French, D. Mareschal, E. Thomas: Learning to perceive time: A connectionist, memory-decay model of the development of interval timing in infants. In: Proc. 33rd Annu. Conf. Cogn. Sci. Soc. (COGSCI) (2011)Google Scholar
  19. 37.19
    P. Craenen: Instruments for new ears, ISCM World Music Mag. 22, 90–99 (2012)Google Scholar
  20. 37.20
    M. Leman, L. Nijs: Music cognition and technology-enhanced learning for music playing. In: The Routledge Companion to Music, Technology & Education, ed. by A. King, A. Ruthmann, E. Himonides (Routledge, New York 2015)Google Scholar
  21. 37.21
    F. Bevilacqua, B. Zamborlin, A. Sypniewski, N. Schnell, F. Guédy, N. Rasamimanana: Continuous realtime gesture following and recognition. In: Embodied Communication and Human-Computer Interaction, Vol. 5934, ed. by S. Kopp, I. Wachsmuth (Springer, Berlin, Heidelberg 2010)Google Scholar
  22. 37.22
    P.-J. Maes, M. Leman, C. Palmer, M.M. Wanderley: Action-based effects on music perception, Front. Psychol. 4, 1008 (2013)Google Scholar
  23. 37.23
    G. Knoblich, N. Sebanz: The social nature of perception and action, Curr. Dir. Psychol. Sci. 15(3), 99–104 (2006)CrossRefGoogle Scholar
  24. 37.24
    W. Schultz: The reward signal of midbrain dopamine neurons, Physiology 14(6), 249–255 (1999)CrossRefGoogle Scholar
  25. 37.25
    R.A. Wise: Dopamine, learning and motivation, Nat. Rev. Neurosci. 5(6), 483–494 (2004)CrossRefGoogle Scholar
  26. 37.26
    P.C. Fletcher, J.M. Anderson, D.R. Shanks, R. Honey, T.A. Carpenter, T. Donovan, E.T. Bullmore: Responses of human frontal cortex to surprising events are predicted by formal associative learning theory, Nature Neurosci. 4(10), 1043–1048 (2001)CrossRefGoogle Scholar
  27. 37.27
    M.V. Puig, J. Rose, R. Schmidt, N. Freund: Dopamine modulation of learning and memory in the prefrontal cortex: Insights from studies in primates, rodents, and birds, Front. Neural Circuits 8, 93 (2014)CrossRefGoogle Scholar
  28. 37.28
    W. Schultz: Behavioral dopamine signals, Trends Neurosci. 30(5), 203–210 (2007)CrossRefGoogle Scholar
  29. 37.29
    E.E. Steinberg, R. Keiflin, J.R. Boivin, I.B. Witten, K. Deisseroth, P.H. Janak: A causal link between prediction errors, dopamine neurons and learning, Nat. Neurosci. 16(7), 966–973 (2013)CrossRefGoogle Scholar
  30. 37.30
    P. Waelti, A. Dickinson, W. Schultz: Dopamine responses comply with basic assumptions of formal learning theory, Nature 412(6842), 43–48 (2001)CrossRefGoogle Scholar
  31. 37.31
    T.E. Hazy, M.J. Frank, R.C. O’Reilly: Neural mechanisms of acquired phasic dopamine responses in learning, Neurosci. Biobehav. Rev. 34(5), 701–720 (2010)CrossRefGoogle Scholar
  32. 37.32
    W. Schultz: Predictive reward signal of dopamine neurons, J. Neurophysiol. 80(1), 1–27 (1998)MathSciNetCrossRefGoogle Scholar
  33. 37.33
    R.J. Zatorre, V.N. Salimpoor: From perception to pleasure: Music and its neural substrates, Proc. Natl. Acad. Sci. 110(Supplement 2), 10430–10437 (2013)CrossRefGoogle Scholar
  34. 37.34
    I. Peretz, M. Coltheart: Modularity of music processing, Nat. Neurosci. 6(7), 688–691 (2003)CrossRefGoogle Scholar
  35. 37.35
    D.B. Huron: Sweet Anticipation: Music and the Psychology of Expectation (MIT Press, Cambridge 2006)Google Scholar
  36. 37.36
    P. Vuust, M.L. Kringelbach: The pleasure of making sense of music, Interdiscip. Sci. Rev. 35(2), 166–182 (2010)CrossRefGoogle Scholar
  37. 37.37
    B. Hommel: Acquisition and control of voluntary action. In: Voluntary Action: Brains, Minds, and Sociality, ed. by S. Maasen, W. Prinz, G. Roth (Oxford Univ. Press, Oxford 2003) pp. 34–48Google Scholar
  38. 37.38
    G. Rizzolatti, L. Craighero: Mirror neuron: A neurological approach to empathy. In: Neurobiology of Human Values, ed. by J.-P.P. Changeux, A. Damasio, W. Singer (Springer, Berlin, Heidelberg 2005) pp. 107–123CrossRefGoogle Scholar
  39. 37.39
    N. Bryan-Kinns: Mutual engagement in social music making. In: Intelligent Technologies for Interactive Entertainment, ed. by A. Camurri, C. Costa (Springer, Berlin, Heidelberg 2012) pp. 260–266CrossRefGoogle Scholar
  40. 37.40
    N. Moran: Social implications arise in embodied music cognition research which can counter musicological ‘‘individualism’’, Front. Psychol. 5, 676 (2014)CrossRefGoogle Scholar
  41. 37.41
    L. Badino, A. D’Ausilio, D. Glowinski, A. Camurri, L. Fadiga: Sensorimotor communication in professional quartets, Neuropsychologia 55, 98–104 (2014)CrossRefGoogle Scholar
  42. 37.42
    D. Glowinski, M. Mancini, A. Camurri: Studying the effect of creative joint action on musicians’ behavior. In: Arts and Technology, Vol. 116, ed. by G. De Michelis, F. Tisato, A. Bene, D. Bernini (Springer, Berlin, Heidelberg 2013) pp. 113–119CrossRefGoogle Scholar
  43. 37.43
    M. Leman, M. Demey, M. Lesaffre, L. van Noorden, D. Moelants: Concepts, technology, and assessment of the social music game ’Sync-in-Team’. In: Proc. Int. Conf. Comput. Sci. Eng. (CSE) (2009)Google Scholar
  44. 37.44
    L. Shapiro: The Routledge Handbook of Embodied Cognition (Routledge, New York 2014)Google Scholar
  45. 37.45
    G.F. Welch, D.M. Howard, E. Himonides, J. Brereton: Real-time feedback in the singing studio: An innovatory action-research project using new voice technology, Music Educ. Res. 7(2), 225–249 (2005)CrossRefGoogle Scholar
  46. 37.46
    G. Riva: Enacting interactivity: The role of presence. In: Enacting Intersubjectivity: A Cognitive and Social Perspective on the Study of Interactions, ed. by F. Morganti, A. Carassa, G. Riva (IOS Press, Amsterdam 2008) pp. 97–114Google Scholar
  47. 37.47
    G. Riva, G. Castelnuovo, F. Mantovani: Transformation of flow in rehabilitation: The role of advanced communication technologies, Behav. Res. Methods 38(2), 237–244 (2006)CrossRefGoogle Scholar
  48. 37.48
    J. Russell: Agency: Its Role in Mental Development (Psychology, Hove 1996)Google Scholar
  49. 37.49
    T. Metzinger: Phenomenal transparency and cognitive self-reference, Phenomenol. Cogn. Sci. 2(4), 353–393 (2003)CrossRefGoogle Scholar
  50. 37.50
    G. Dubus, R. Bresin: A systematic review of mapping strategies for the sonification of physical quantities, PLoS ONE 8(12), e82491 (2013)CrossRefGoogle Scholar
  51. 37.51
    R. Sigrist, G. Rauter, R. Riener, P. Wolf: Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review, Psychon. Bull. Rev. 20(1), 21–53 (2013)CrossRefGoogle Scholar
  52. 37.52
    H. Haken, J.A.S. Kelso, H. Bunz: A theoretical model of phase transitions in human hand movements, Biol. Cybern. 51(5), 347–356 (1985)MathSciNetCrossRefGoogle Scholar
  53. 37.53
    J.A.S. Kelso: Coordination dynamics. In: Encyclopedia of Complexity and Systems Science, ed. by R.A. Meyers (Springer, Berlin, Heidelberg 2009) pp. 1537–1565CrossRefGoogle Scholar
  54. 37.54
    O. Oullier, G.C. De Guzman, K.J. Jantzen, J. Lagarde, J.A.S. Kelso: Social coordination dynamics: Measuring human bonding, Soc. Neurosci. 3(2), 178–192 (2008)CrossRefGoogle Scholar
  55. 37.55
    B. Moens, M. Leman: Alignment strategies for the entrainment of music and movement rhythms, Ann. N.Y. Acad. Sci. 1337, 86–93 (2015)CrossRefGoogle Scholar
  56. 37.56
    B. Moens, C. Muller, L. van Noorden, M. Franek, B. Celie, J. Boone, M. Leman: Encouraging spontaneous synchronisation with D-Jogger, an adaptive music player that aligns movement and music, PLoS ONE 9(12), e114234 (2014)CrossRefGoogle Scholar
  57. 37.57
    K. Friston, J. Kilner, L. Harrison: A free energy principle for the brain, J. Physiol. Paris 100(1), 70–87 (2006)CrossRefGoogle Scholar
  58. 37.58
    M.I. Jordan, D.E. Rumelhart: Forward models: Supervised learning with a distal teacher, Cogn. Sci. 16(3), 307–354 (1992)CrossRefGoogle Scholar
  59. 37.59
    J.W. Krakauer, P. Mazzoni: Human sensorimotor learning: Adaptation, skill, and beyond, Curr. Opin. Neurobiol. 21(4), 636–644 (2011)CrossRefGoogle Scholar
  60. 37.60
    H. Lalazar, E. Vaadia: Neural basis of sensorimotor learning: Modifying internal models, Curr. Opin. Neurobiol. 18(6), 573–581 (2008)CrossRefGoogle Scholar
  61. 37.61
    R. Shadmehr, M.A. Smith, J.W. Krakauer: Error correction, sensory prediction, and adaptation in motor control, Annu. Rev. Neurosci. 33, 89–108 (2010)CrossRefGoogle Scholar
  62. 37.62
    M.C.M. van der Steen, P.E. Keller: The ADaptation and Anticipation Model (ADAM) of sensorimotor synchronization, Front. Hum. Neurosci. 7, 253 (2013)Google Scholar
  63. 37.63
    D.M. Wolpert, Z. Ghahramani, M.I. Jordan: An internal model for sensorimotor integration, Sci. New Ser. 269(5232), 1880–1882 (1995)Google Scholar
  64. 37.64
    L. Nijs, M. Leman: Interactive technologies in the instrumental music classroom: A longitudinal study with the Music Paint Machine, Comput. Educ. 73, 40–59 (2014)CrossRefGoogle Scholar
  65. 37.65
    W. Hsu, M. Sosnick: Evaluating interactive music systems: An HCI approach. In: Proc. Int. Conf. New Interf. Music. Expr. (NIME) (2009)Google Scholar
  66. 37.66
    C. Kiefer, N. Collins, G. Fitzpatrick: HCI methodology for evaluating musical controllers: A case study. In: Proc. Int. Conf. New Interf. Music. Expr. (NIME) (2008)Google Scholar
  67. 37.67
    M.M. Wanderley, N. Orio: Evaluation of input devices for musical expression: Borrowing tools from HCI, Comput. Music J. 26(3), 62–76 (2002)CrossRefGoogle Scholar
  68. 37.68
    A.R. Addessi, F. Ferrari, F. Carugati: Observing and measuring the flow emotional state in children interacting with the MIROR platform. In: 12th Int. Conf. Music Percept. Cogn. (ICMPC)/8th Trienn. Conf. Eur. Soc. Cogn. Sci. Music (ESCOM), Thessaloniki (2012)Google Scholar
  69. 37.69
    L. Nijs, P. Coussement, B. Moens, D. Amelinck, M. Lesaffre, M. Leman: Interacting with the Music Paint Machine: Relating the constructs of flow experience and presence, Interact. Comput. 24(4), 237–250 (2012)CrossRefGoogle Scholar
  70. 37.70
    N. Collins, J. d’Escriván: The Cambridge Companion to Electronic Music (Cambridge Univ. Press, Cambridge 2007)CrossRefGoogle Scholar
  71. 37.71
    L. Nijs, B. Moens, M. Lesaffre, M. Leman: The Music Paint Machine: Stimulating self-monitoring through the generation of creative visual output using a technology-enhanced learning tool, J. New Music Res. 41(1), 79–101 (2012)CrossRefGoogle Scholar
  72. 37.72
    L. Nijs, M. Leman: Performing with the Music Paint Machine: Provoking an embodied approach to educational technology. In: Music, Technology & Education: Critical Perspectives, ed. by A. King, E. Himonides (Routledge, London 2014)Google Scholar
  73. 37.73
    G. Pezzulo: Grounding procedural and declarative knowledge in sensorimotor anticipation, Mind Lang. 26(1), 78–114 (2011)CrossRefGoogle Scholar
  74. 37.74
    M. Csikszentmihalyi: Flow: The Psychology of Optimal Experience, Vol. 41 (Harper Perennial, New York 1991)Google Scholar
  75. 37.75
    C. Peifer: Psychophysiological correlates of flow-experience. In: Advances in Flow Research, ed. by S. Engeser (Springer, Berlin, Heidelberg 2012) pp. 139–164CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2018

Authors and Affiliations

  • Pieter-Jan Maes
    • 1
  • Luc Nijs
    • 2
  • Marc Leman
    • 3
  1. 1.IPEM – Musicology, Department of Art, Music and Theatre SciencesGhent UniversityGhentBelgium
  2. 2.IPEM – Musicology, Department of Art, Music and Theatre SciencesGhent UniversityGhentBelgium
  3. 3.IPEM – Musicology, Department of Art, Music and Theatre SciencesGhent UniversityGhentBelgium

Personalised recommendations