2007: Wireless Sensor Interface and Gesture-Follower for Music Pedagogy

Abstract

We present in this paper a complete gestural interface built to support music pedagogy. The development of this prototype concerned both hardware and software components: a small wireless sensor interface including accelerometers and gyroscopes, and an analysis system enabling gesture following and recognition. A first set of experiments was conducted with teenagers in a music theory class. The preliminary results were encouraging concerning the suitability of these developments in music education.

Appendices

Author Commentary: Once Upon A NIME

Frederic Bevilacqua and Norbert Schnell

The initial aim of this article was to give an overview of research on music pedagogy with tangible interfaces that we were starting at that time in our team. The article contains three fairly independent contributions that alternatively could have been separated into three different articles:

• Wireless sensing hardware

• Gesture analysis and interactive machine learning software

• Applications and use cases in music pedagogy

However, the idea was to describe how different streams of NIME research converged in specific use cases that had been actually implemented. Revisiting the three themes almost 10 years later gives the opportunity to contextualize this work in the flow of still ongoing research and development.

Firstly, the possibility to create a small size wireless sensor interface using off-the-shelf wireless transmission modules marked a considerable breakthrough compared to other systems we reported earlier in the NIME community (see Fléty and Maestracci 2011 and references herein). The advantage of these modules resided in the favorable compromise between a small size, low power consumption and relatively high bandwidth. Moreover, it was possible to use several wireless interfaces in parallel. This matched well with our applications that included music and dance performance as well as experimental music pedagogy. This hardware development, among others at the time, was representative of the increased use of wireless technology at NIME. For our research group, this development allowed us to boost the use of miniature wireless interfaces using inertial measurement units, such as accelerometers and gyroscopes. Since then, we have employed different generations of such IMU-based wireless interface in many applications that more recently also include mobile and web platforms. In fact, the development described in the article coincides with that of the first generation of smartphones as well as the first generation of game controllers that include wireless motion sensing such as the Wiimote (with a much bigger form factor and still lacking precision and reliability).

Secondly, this article featured the first complete description of the “gesture follower,” which has represented an important line of research in our team up until now. The gesture follower has been used since this reports in a large number of artistic performances and installations, in music and dance (Bevilacqua et al. 2011). More complete descriptions followed in subsequent articles and this research influenced similar methods such as “Mapping by Listening” and tools such as GVF (Caramiaux et al. 2014) or even more recently XMM (Françoise et al. 2014). The gesture follower can be considered as our first development in what we call now more generally “interactive machine learning” (but the name was not as common in NIME at that time). In particular, it allowed users to record, as many time they wanted, their gestures to build movement-sound interactions based on rather few examples. We found this flexibility a key element for the music pedagogy use cases we reported on as well as for many other use cases we developed over the past years. Finally, the applications in music education described in the article occupied us for several years of continued research. These “real-world” applications certainly contributed to foster several concepts we’re pursuing, such as the use of “metaphors” and “playing techniques.”

Globally, this proceeding can be seen as the start of a line of research that produced our interface MO—Modular Musical Objects, that were also reported in the NIME community (Schnell et al. 2011) and that also won the Margaret Guthman Musical Instrument Competition.⁹ Many of the ideas and techniques that were presented in this article are still actively pursued, in particular with mobile and web technologies.

Expert Commentary: Gesture Following Made Accessible

Kristian Nymoen

One of the beauties of the NIME conference is the encouragement of publications that focus on novel systems for musical expression. As a result, many NIME publications are often broad in scope, presenting an entire system, including the hardware, software, modes of interaction, musical output, and more. Sometimes the presented system as a whole persists as a symbolic interface for musical expression for many years to come. In other cases a particular sub-unit of the system, such as a piece of hardware or a specific mode of interaction, is the contribution that makes the publication leave its mark within and beyond the community.

The paper by Bevilacqua et al. demonstrates brilliantly how NIME developments may provide important outcomes. The paper is “disguised” as a well-written, yet quite ordinary, NIME paper on a prototype of a complete system for music pedagogy. The custom-made hardware was thoroughly documented and a cutting edge solution for portable sensing and wireless communication. The proposed problem of using technology for studying and teaching fluidity in conducting is still relevant, and the authors showed the applicability of the system for this task through testing and evaluating in a real pedagogical context with students and their teacher. Still, it is the software part of the system that really stands out as one of the important NIME contributions of the decade: the “Gesture Follower.”

The Gesture Follower was implemented in IRCAM’s FTM framework for Max. The FTM framework extends Max with various types of data structures and operators, and facilitates many types of data processing in Max, for instance of motion data. The implementation in Max is one of the main reasons why the Gesture Follower became such an important piece of software. Gesture recognition in music had already been explored for several years at the time of this paper’s publication. However, its application required knowledge of the machine learning algorithms involved and in most cases also proficiency in some text-based programming environment. The out-of-the-box examples and tutorials for the Gesture Follower made gesture recognition accessible to a larger user-group, including musicians and artists who preferred the graphical interface of Max to text-based languages.

Not only was the Gesture Follower more accessible due to its Max implementation, it provided a combination of highly useful features for use in musical interaction. Classification happens in real time, continuously updating the classification score against each of the pre-trained examples while the user is moving. As such, the system can easily be used in mapping between motion data and synthesizer parameters. To allow for different durations between training examples and the input gestures, an elegant time-warping solution has been implemented. For each of the pre-trained examples, the Gesture Follower provides an estimation of the time index within the gesture. In other words, the system does not only recognize the gesture, it follows the gesture. With such a time warping function in place, it is possible match the playback duration to the duration of the input gesture.

Since this first publication on the Gesture Follower, the IRCAM team has presented a number of follow-up articles with various improvements and testing of the system in different contexts (Bevilacqua et al. 2010, 2012). One of the developments is the implementation of the Gesture Follower in IRCAM’s multi-buffer solution for audio and motion capture data in Max, Mubu, with improvements in both functionality and user interface.

The paper of Bevilacqua et al. shows that developing new interfaces for musical expression is more than just developing a system in itself. NIME technologies are often highly generic, and may have much broader application areas than the system that is presented in the NIME publication. In this specific case, the Gesture Follower was presented as one part of a system for music pedagogy, but has proven just as useful in music performance (Van Nort et al. 2013), specialized systems for multimodal score-following (Ritter et al. 2013), or even tasks unrelated to music, such as recognition of wheel-throwing pottery gestures (Manitsaris et al. 2014).