Improvising with Digital Auto-Scaffolding: How Mimi Changes and Enhances the Creative Process

Abstract

This chapter poses, and proposes some answers to, questions about the origins and nature of creativity when digital media takes an active role in the music-making process. The discussions are centered on François’ Mimi (Multimodal Interaction for Musical Improvisation ) system, which enables a musician to seed the computer with musical ideas and then improvise atop re-combinations of these ideas; the system provides the musician with visual foreknowledge of the machine’s intent and review of the interaction. They extend to the different instantiations of, and extensions to, the Mimi system, which are designed with various interaction nuances in mind, and engender new forms of creativity. We review each Mimi version, from the original blue-and-white silhouette display, to the Scriabin -inspired varicolored panels, to the multi-paneled user-directed Mimi4x . In each scenario, we consider the impact of Mimi on the creative process and the resulting performance; specifically, we describe the interaction between a performer, the composer (when this is different from the performer), and the system, analyzing the techniques used to successfully negotiate a performance with Mimi, and the formal musical structures that result from this interaction.

This chapter incorporates, in part and in modified form, material that has previously appeared in “Mimi4x: An Interactive Audio-visual Installation for High-level Structural Improvisation” (Alexandre R. J. François, Isaac Schankler and Elaine Chew, International Journal of Arts and Technology, vol. 6, no. 2, 2013), “Performer Centered Visual Feedback for Human-Machine Improvisation” (Alexandre R. J. François, Elaine Chew and Dennis Thurmond, ACM Computers in Entertainment, vol. 9, no. 3, November 2011, 13 pages), “Preparing for the Unpredictable: Identifying Successful Performance Strategies in Human-Machine Improvisation” (Isaac Schankler, Alexandre R. J. François and Elaine Chew, Proceedings of the International Symposium on Performance Science, Toronto, Canada, 24–27 August 2011), and “Emergent Formal Structures of Factor Oracle-Driven Musical Improvisations” (Isaac Schankler, Jordan L.B. Smith, Alexandre R. J. François and Elaine Chew, Proceedings of the International Conference on Mathematics and Computation in Music, Paris, France, 15–17 June 2011).