Abstract
Single- and multi-agent installations and performances that use physiological signals to establish an interface between music and mental states can be found as early as the mid-1960s. Among these works, many have used physiological signals (or inferred cognitive, sensorimotor or affective states) as media for music generation and creative expression. To a lesser extent, some have been developed to illustrate and study effects of music on the brain. Historically, installations designed for a single participant are most prevalent. Less common are installations that invite participation and interaction between multiple individuals. Implementing such multi-agent installations raises unique challenges, but also unique possibilities for social interaction. Advances in unobtrusive and/or mobile devices for physiological data acquisition and signal processing, as well as computational methods for inferring mental states from such data, have expanded the possibilities for real-world, multi-agent, brain–music interfaces. In this chapter, we examine a diverse selection of playful and social installations and performances, which explore relationships between music and the brain and have featured publically in Mainly Mozart’s annual Mozart & the Mind (MATM) festival in San Diego. Several of these installations leverage neurotechnology (typically novel wearable devices) to infer brain states of participants. However, we also consider installations that solely measure behavior as a means of inferring cognitive state or to illustrate a principle of brain function. In addition to brief overviews of implementation details, we consider ways in which such installations can be useful vehicles, not only for creative expression, but also for education, social interaction, therapeutic intervention, scientific and aesthetic research, and as playful vehicles for exploring human–human and human–machine interaction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Processing code available at (http://www.openprocessing.org/sketch/174919) and was adapted from the sketch “Network Excitation” (http://www.openprocessing.org/sketch/63796).
References
Alivisatos, A.P., Chun, M., Church, G.M., Greenspan, R.J., Roukes, M.L., Yuste, R.: The brain activity map project and the challenge of functional connectomics. Neuron 74, 970–974 (2012)
Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi, O., Faraji, F., Janowich, J., Kong, E., Laraburro, Y., Rolle, C., Johnston, E., Gazzaley, A.: Video game training enhances cognitive control in older adults. Nature 501, 97–101 (2013)
Aspell, J.E., Heydrich, L., Marillier, G., Lavanchy, T., Herbelin, B., Blanke, O.: Turning body and self inside out: visualized heartbeats alter bodily self-consciousness and tactile perception. Psychol. Sci. 24(12), 2445–2453 (2013)
Broughton, M., Stevens, C.: Music, movement and marimba: an investigation of the role of movement and gesture in communicating musical expression to an audience. Psychol. Music 37(2), 137–153 (2009)
DeNora, T.: Music in Everyday Life. Cambridge University Press, Cambridge (2000)
Duvinage, M., Castermans, T., Dutoit, T.: A P300-based quantitative comparison between the Emotiv EPOC headset and a medical EEG device. Biomed. Eng. Online (2012). doi:10.1186/1475-925X-12-56
Fisher, N.I.: Statistical Analysis of Circular Data. Cambridge University Press, Cambridge (1993)
Glowinski, D., Riolfo, A., Shirole, K., Torres-Eliard, K., Chiorri, C., Grandjean, D.: Is he playing solo or within an ensemble? How the context, visual information, and expertise may impact upon the perception of musical expressivity. Perception 43(8), 825–828 (2014)
Goldberg, J.M., Brown, P.B.: Functional organization of the dog superior olivary complex: an anatomical and electrophysiological study. J. Neurophysiol. 31, 639–656 (1968)
Grayson, J. (ed.): Sound Sculpture. Aesthetic Research Centre of Canada Publications, Vancouver (1975)
Greenfield, A.: Everyware: The Dawning Age of Ubiquitous Computing, 1st edn, 272p. New Riders Publishing, USA (2006). ISBN 0-321-38401-6
Gurevich, M.A., Fyans, C.: Digital musical interactions: Performer–system relationships and their perception by spectators. Organised Sound. 16(2), 166–175 (2011)
Hagmann, P., Cammoun, L., Gigandet, X., Meuli, R., Honey, C.J., Wedeen, V.J., Sporns, O., Friston, K.J.: Mapping the structural core of human cerebral cortex. PLoS Biol. 6, e159 (2008)
He, Y., Wang, J., Wang, L., Chen, Z.J., Yan, C., Yang, H., Tang, H., Zhu, C., Gong, Q., Zang, Y., Evans, A.C.: Uncovering intrinsic modular organization of spontaneous brain activity in humans. PLoS ONE 4, e5226 (2009)
Henry, T.K.: Invention locates hurt brain cells. New York Times, p. 21, 2 March 1943
Herholz, S., Zatorre, R.: Musical training as a framework for brain plasticity: behavior, function, and structure. Neuron 76(3): 486–502 (2012). ISSN 0896-6273
Hettinger, L.J., Berbaum, K.S., Kennedy, R.S., Dunlap, W.P., Nolan, M.D.: Vection and simulator sickness. Mil. Psychol. 2(3), 171–181 (1990)
Iversen, J.R., Patel, A.D.: The beat alignment test (BAT): surveying beat processing abilities in the general population. In: Ken’ichi, M., Yuzuru, H., Mayumi, A., Yoshitaka, N., Minoru, T. (eds.) Proceedings of the 10th International Conference on Music Perception and Cognition (ICMPC10) Sapporo, Japan, pp. 465–468 (2008)
Khalil, A.K., Minces, V., McLoughlin, G., Chiba, A.: Group rhythmic synchrony and attention in children. Front. Psychol. 4, 564 (2013)
Koelsch, S.: Toward a neural basis of music perception—a review and updated model. Front. Psychol. 2, 110 (2011)
Koelsch, S., Siebel, W.: Towards a neural basis of music perception. Trends Cogn. Sci. 9(12), 578–584 (2005)
Leslie, G., Mullen, T.: MoodMixer: EEG-based collaborative sonification. In: Jensenius, A.R., Tveit, A., Godøy, R.I., Overholt, D. (eds.) Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 296–299 (2011). ISBN: 978-82-991841-7-5
Lin, Y., Duann, J., Chen, J., Jung, T.-P.: Electroencephalographic dynamics of musical emotion perception revealed by independent spectral components. NeuroReport 21(6), 410 (2010)
Loui, P., Koplin-Green, M., Frick, M., Massone, M.: Rapidly learned identification of epileptic seizures from sonified EEG. Front. Hum. Neurosci. 8, 820 (2014)
Lucier, A.: Reflections: Interviews, Scores, Writings. MusikTexte, Koln (1995)
Mann, S., Fung, J., Garten, A.: DECONcert: bathing in the light, sounds, and waters of the musical brainbaths. In: Proceedings of the 2007 International Computer Music Conference (ICMC2007), vol. 2, pp. 204–211, Copenhagen, Denmark, 27–31 August 2007
McNeill, W.H. Keeping Together in Time: Dance and Drill in Human History. Harvard University Press, Cambridge (1997)
Miranda, E.R.: Brain–Computer music interfacing: interdisciplinary research at the crossroads of music, science and biomedical engineering. In: Miranda, E.R., Castet, J. (eds.) Guide to Brain-Computer Music Interfacing, pp. 1–27. Springer, London (2014)
Montague, P.R., Berns, G.S., Cohen, J.D., et al.: Hyperscanning: simultaneous fMRI during linked social interactions. NeuroImage 16(4), 1159–1164 (2002)
Mullen, T.R.: The dynamic brain: modeling neural dynamics and interactions from human electrophysiological recordings, 446 pp. Dissertation, University of California, San Diego (2014)
Mullen, T., Worrell, G., Makeig, S.: Multivariate principal oscillation pattern analysis of ICA sources during seizure. In: Proceedings of the 34th Annual International Conference of the IEEE, EMBS, San Diego, CA (2012)
Mullen, T., Kothe, C., Konings, O., Gazzaley, A.: Real-time functional brain imaging: how GPU acceleration redefines each stage. In: GPU Technology Conference, GTC 2014—ID S4633, 26 March 2014. http://on-demand-gtc.gputechconf.com/gtcnew/on-demand-gtc.php#sthash.9dVqqGnV.dpuf (2014)
Müller, V., Sänger, J., Lindenberger, U.: Intra- and inter-brain synchronization during musical improvisation on the guitar. PLoS ONE 8(9), e73852 (2013)
Nijholt, A.: Competing and collaborating brains: multi-brain computer interfacing. In: Hassanieu, A.E., Azar, A.T. (eds.) Brain–Computer interfaces: current trends and applications, vol. 74, pp. 313–335. Springer International Publishing, Switzerland (2015)
Patel, A.D., Iversen, J.R.: The evolutionary neuroscience of musical beat perception: the action simulation for auditory prediction (ASAP) hypothesis. Front. Syst. Neurosci. 8, 1–31 (2014)
Rosenboom, D. (ed.): Biofeedback and the Arts, Results of Early Experiments. Aesthetic Research Centre of Canada Publications, Vancouver (1976)
Rosenboom, D.: Interactive music with intelligent instruments—a new, propositional music? In: Brooks, E. (ed.) New Music Across America, pp. 66–70. California Institute of the Arts and High Performance Books, Valencia and Santa Monica, CA (1992)
Rosenboom, D.: Extended musical interface with the human nervous system: assessment and prospectus. Revised electronic monograph: http://www.davidrosenboom.com/media/extended-musical-interface-human-nervous-system-assessment-and-prospectus (1997) (Original (1990), San Francisco: Leonardo Monograph Series, 1)
Rosenboom, D.: Extended musical interface with the human nervous system: assessment and prospectus. Leonardo 32(4), 257–259 (1999)
Rosenboom, D.: Invisible gold, classics of live electronic music involving extended musical interface with the human nervous system. Audio CD, p. 21022-2. Pogus Productions, Chester, New York) (2000)
Rosenboom, D.: Propositional music from extended musical interface with the human nervous system. In: Avanzini, G. et al. (eds.) The Neurosciences and Music, Annals of the New York Academy of Sciences, vol. 999, pp. 263–271. New York Academy of Sciences, New York (2003)
Rosenboom, D.: Brainwave music 2006. Audio CD. EM Records #EN1054CD, Osaka, Japan (2006)
Sänger, J., Müller, V., Lindenberger, U.: Intra- and inter-brain synchronization and network properties when playing guitar in duets. Front. Hum. Neurosci. 6, 312 (2012)
Song, Y., Dixon, S., Pearce, M.: A survey of music recommendation systems and future perspectives. In: 9th International Symposium on Computer Music Modeling and Retrieval (2012)
Ueno, K., Kato, K., Kawai, K.: Effect of room acoustics on musicians’ performance. Part I: experimental investigation with a conceptual model. Acta Acustica United Acustica 96(3), 505–515 (2010)
Wang, Y., Jung, T.-P.: A collaborative brain–computer interface for improving human performance. PLoS ONE 6(5), e20422 (2011)
Yun, K., Watanabe, K., Shimojo, S.: Interpersonal body and neural synchronization as a marker of implicit social interaction. Sci. Rep. 2, 959 (2012)
Zander, T., Kothe, C., Jatsev, S., Gaertner, M.: Enhancing human–computer interaction with input from active and passive brain–computer interfaces. In: Brain-Computer Interfaces. Human-Computer Interaction Series, pp. 181–199 (2010)
Acknowledgments
We gratefully acknowledge ViaSat (San Diego) for its generous sponsorship of Mozart and the Mind. We further acknowledge Nancy Laturno Bojanic and the entire staff at Mainly Mozart for their assistance in making these installations possible. We thank Cognionics Inc. for donating the wearable EEG equipment for The Floating Man, NeuroDrummer/GlassBrain, and Ringing Minds. We are also grateful to the following institutions for their contributions of equipment or personnel: Nvidia, Syntrogi Inc, Remo Inc, Resounding Joy, Mindo, InteraXon, and the Swartz Center for Computational Neuroscience at UC San Diego. Additionally, R. Warp thanks John D. Long, Joyce Shoyi Golomb (Emotiv Systems), Belinda Reynolds, Tim Mullen, and Erica Warp for their contributions to Spukhafte Fernwirkung. G. Leslie and T. Mullen thank Maxwell Citron for his contribution to Four Stream Mind used in MoodMixer 3.0. M. Whitman thanks Michael Gonzales for his contribution to NeuroDrummer. T. Ward thanks Allen Gruber, Tim Mullen, and Mike Chi for their contributions to The Floating Man. T. Mullen thanks Christian Kothe and Mike Chi for their contributions to Ringing Minds. Photographic credit for MATM goes to Katarzyna Woronowicz (jkatphoto.com) for Mainly Mozart.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer Science+Business Media Singapore
About this chapter
Cite this chapter
Mullen, T. et al. (2015). MindMusic: Playful and Social Installations at the Interface Between Music and the Brain. In: Nijholt, A. (eds) More Playful User Interfaces. Gaming Media and Social Effects. Springer, Singapore. https://doi.org/10.1007/978-981-287-546-4_9
Download citation
DOI: https://doi.org/10.1007/978-981-287-546-4_9
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-287-545-7
Online ISBN: 978-981-287-546-4
eBook Packages: EngineeringEngineering (R0)