Design of Vibrotactile Feedback and Stimulation for Music Performance
- 7.6k Downloads
Haptics, and specifically vibrotactile-augmented interfaces, have been the object of much research in the music technology domain: In the last few decades, many musical haptic interfaces have been designed and used to teach, perform, and compose music. The investigation of the design of meaningful ways to convey musical information via the sense of touch is a paramount step toward achieving truly transparent haptic-augmented interfaces for music performance and practice, and in this chapter we present our recent work in this context. We start by defining a model for haptic-augmented interfaces for music, and a taxonomy of vibrotactile feedback and stimulation, which we use to categorize a brief literature review on the topic. We then present the design and evaluation of a haptic language of cues in the form of tactile icons delivered via vibrotactile-equipped wearable garments. This language constitutes the base of a “wearable score” used in music performance and practice. We provide design guidelines for our tactile icons and user-based evaluations to assess their effectiveness in delivering musical information and report on the system’s implementation in a live musical performance.
KeywordsVibrotactile Feedback Musical Performance Tactile Icons Wear Scores Tactons
In recent years, the widespread availability of smartphones and tablet computers made vibrotactile technology—in the form of actuators specifically designed to stimulate a user’s sense of touch via vibration—inexpensive and readily available. Haptic researchers, both in academic and industrial contexts, have been designing ways of communicating via the sense of touch by means of tactile effects used to provide information such as: navigational cues , textures , or notifications . Systematic studies have been conducted to assess the efficiency of these effects in well-defined contexts, and new prototypes and applications are constantly being investigated.
In the music domain, the sense of touch can be used to convey relevant musical information, such as articulation  and timing , especially in professional performances . Several haptic interfaces for music performance and practice have been created in the last two decades, but for very few of these a thorough evaluation of their effectiveness has been conducted.
In this chapter, we present our work in the development and preliminary evaluation of meaningful ways to provide information to performers via the sense of touch for music performance and practice. Our research, conducted in the context of a multidisciplinary project involving haptic researchers, composers, and wearable designers, is aimed at the development of a language of tactile icons specifically designed to convey musical information to professional musicians. These icons, delivered via specialized garments equipped with arrays of vibrotactile actuators, have been evaluated to determine their effectiveness and reliability. They will be used as the building blocks of a wearable score language, which composers will use to create new pieces and art installations.
To provide a theoretical framework for this research, we present a brief overview of the current state of haptic feedback and stimulation in music technology. We expand the classical models of digital musical instruments (DMIs)  to include general-purpose tactile interfaces, i.e., devices where other sensory feedback may not be present and tactile feedback can be arbitrary mapped to external sources of information. We then present a literature review together with a taxonomy of tactile feedback and stimulation. This categorization is aimed at emphasizing the different functional roles that haptic technology can achieve in conveying musically relevant information.
10.2 Haptic Feedback in Music Technology
Haptic technology has been widely used in the development of interfaces for musical expression and musical interaction, and two main classes of devices can be identified in this context: DMIs and general-purpose haptic interfaces.
In traditional musical instruments, the tactile and kinaesthetic feedback coming from the resonating parts of the instrument give the performer important information about their interaction [1, 20, 28, 43] (see Chap. 2). In DMIs, the decoupling of gesture acquisition from sound synthesis has the important effect of breaking the mechanical feedback loop between performer and sound-producing structures. Haptic feedback becomes then an arbitrary design factor , and the choice of actuators and signals used to drive them (see Sect. 13.2) defines the instrument’s architecture.
Haptic devices can provide tactile cues during performance with DMIs, not only if embedded into the instruments themselves, but also when deployed separately by means of tactile displays and wearable devices that can be used to go beyond the direct performer–instrument interaction. In the context of music performance, these devices, which we refer to as general-purpose haptic interfaces, can convey information about performers’ interactions with a live-electronics system  or as learning tools to direct and guide users’ gestures via vibrotactile feedback  (see also Chap. 11). They can also be used to convey score cues to a performer on stage  by means of abstract languages of tactile icons . In this context, the distinction between feedback and stimulation becomes clear: The former is a direct response of the instrument or the general-purpose interface to a user’s action; the latter is not issued from a player–device interaction, but it is a means of communication with the user, mediated by the tactile actuators in the interface, which can be used to convey any sort of information.
These displays usually provide either localized (i.e., single body site) or distributed vibrations (via actuators placed on multiple body sites), requiring the design of tactile effects more centered on temporal or spatial properties, respectively, or a combination of both.
10.2.1 Models of Haptic-Enabled Interfaces
The relationship between performer, haptic-enabled musical interface (either general-purpose device or DMI), and audience can be complex, and a number of abstract models of the interaction between these components can be found in the literature. In the case of DMIs several models exist, each of which emphasizes different aspects of the instrument’s design. Marshall  reviews four of these models [4, 5, 9, 54] and proposes a hybrid model merging characteristics across them.
10.2.2 Haptic-Enabled Interfaces
Haptic-enabled interfaces for music performance can be categorized according to the way they deliver haptic feedback and stimulation to the final users. Both DMIs and general-purpose devices can address either the kinaesthetic or the tactile modality, and this can be done in an active or a passive way : Passive feedback and stimulation come from the inherent physical properties of the interface and are not issued by the system’s haptic generator; active interfaces implement a haptic generator to provide user with the designed kinaesthetic and tactile effects.
We will present some of the most important devices present in the literature following these two categories and provide a threefold taxonomy for the active tactile case.
10.2.2.1 Passive Kinaesthetic Feedback
Passive kinaesthetic feedback and stimulation are inherent to the physical characteristics of the controller, and do not require any externally synthesized signal.
O’Modhrain and Essl developed three DMIs that implement passive kinaesthetic feedback. The Pebble Box and the Crumble Bag  were used to control an event-based granular synthesizer: the Pebble Box consists of a box filled with different-sized pebble stones and a microphone that picks up the noise produced by the collisions between pebbles. The kinaesthetic feedback offered by the interface comes from the physical properties of the pebbles themselves, and the impact sounds act as triggering events on the granular synthesizer. The Crumble Bag follows the same patter, and it is aimed to take advantage of natural “grabbing gestures.” A fabric bag is filled with different materials that provide haptic feedback, and a small microphone in the bag provides the necessary event triggers to the algorithm. The Scrubber  also implemented the same approach: an eraser embedded with a force sensor and two microphones were used to control the synthesis of friction sounds, synthesized by means of granular or wavetable synthesis. The haptic feedback again was directly issued by the manipulation of the device dragged along a surface.
Sinyor and Wanderley  developed the Gyroyre, a handheld controller based on a spinning wheel, in which the kinaesthetic feedback comes directly from the dynamic properties of the system. The mapping and synthesis algorithm are designed to take advantage of the haptic feedback, and the interface can be used for different musical applications, sequencing or modifying effects’ parameters.
10.2.2.2 Active Kinaesthetic Interfaces
Active kinaesthetic feedback is the response of the controller to the user’s actions, usually by means of synthesized signals supplied into motors or actuators, which stimulate kinaesthetic receptors. This is most commonly referred to as force feedback.
The earliest example of a force-feedback device specifically developed for musical applications is probably the Transducteur Gestuel Rétroactif (TGR) developed at ACROE, whose development is described in Sect. 8.3. This device was recently used by Sinclair et al.  to investigate velocity estimation methods in the haptic rendering of a bowed string.
Another classical example is the Moose, developed by O’Modhrain and Gillespie , consisting of a plastic puck that the user can manipulate in a 2D space, which is attached to flexible metal bars, connected to linear motors. Two encoders sense the movements of the puck, and the motors provide the correspondent force feedback. The device was used in a bowing test, using a virtual string programmed in Synthesis ToolKit (STK) , where the presence of friction between the bow and the string was simulated using the haptic device.
The vBow by Nichols  is a violin-like controller that uses a series of servomotors and encoders to sense the movement of a rod, acting as the bow, connected to a metallic cable. In its last incarnation, the vBow is capable of sensing moment in 4-DoF and producing haptic feedback accordingly.
More recently, Berdahl and Kontogeorgakopoulos  developed the FireFader, a motorized faders using sensors and DC motors to introduce musicians to haptic controllers. Both the software and hardware used for the project are open-source, allowing musicians to customize the mapping of the interface to their specific needs. Applications of the device are described in Chap. 9.
10.2.2.3 Passive Tactile Interfaces
Passive tactile is a form of primary feedback, which leverages the use of different types of materials in a controller for musical expression. The properties of these materials (e.g., stiffness, flexibility) can affect the ergonomics of the instrument and its feel in the user’s hands.
As an example, the Meta-Instrument  has the form of a partial exoskeleton embedded with buttons that the performer uses to trigger samples and events in the sound; the performer’s gestures are captured via sensors in the arms and mapped to various effects. The buttons embedded in the controller are covered in a layer of foam, providing the user with immediate passive feedback about the level of pressure applied.
10.2.2.4 Active Tactile Feedback and Stimulation: A Taxonomy for Musical Interaction
Active tactile feedback and stimulation are the main focus of this chapter, and for this reason we provide a more in-depth analysis of the related literature, as well as an updated taxonomy, based on Giordano and Wanderley , which will help categorize examples in this field.
We propose a classification identifying in active tactile feedback and stimulation three different categories according to the function that the tactile effects have in the interface design: tactile notification, tactile translation, and tactile languages.
The most straightforward application of tactile stimulation is intended for notifying the users about events taking place in the surrounding environment or about results of their interaction with a system. The effects designed for this kind of applications can be as simple as single, supra-threshold stimuli2 aimed at directing users’ attention, but they can also be more complex, implementing temporal envelopes and/or spatial patterns.
Michailidis and Berweck  and Michailidis and Bullock  have explored solutions to provide haptic feedback in live-electronics performance. The authors developed the Tactile Feedback Tool, a general-purpose interface using small vibrating motors embedded in a glove. The interface gave musicians information about the successful triggering of effects in a live-electronics performance, using an augmented trumpet or a foot pedal switch. This device leverages the capacity of the tactile sense to attract users’ attention, while not requiring them to lose focus on other modalities, which would have been the case with the use of onstage visual displays.
Van der Linden et al.  implemented a whole-body general-purpose vibrotactile device. The authors used a motion capture system and a suit embedded with vibrating motors distributed over the body to enhance the learning process of bowing for novice violin players. A set of ideal bowing trajectories was computed using the motion capture system; when practicing, the players’ postures would be compared in real time with the predefined ideal trajectories. If the distance between any two corresponding points in the two trajectories exceeded the threshold value, the motor spatially closer to that point would vibrate, notifying the users to correct their posture. The authors conducted a study in which several players used the suit during their violin lessons. Results showed an improved coordination of the bowing arm, and participants reported an enhancement in their body awareness produced by the feedback.
A similar solution was developed by Grosshauser and Hermann , which used a vibrating actuator embedded in a violin bow to correct hand posture. Using accelerometers and gyroscopes, the position of the bow could be compared in real time to a given trajectory, and the tactile feedback would automatically activate to notify the users about their wrong posture.
With tactile translation, we refer to two separate classes of applications: One class implements sensory substitution techniques to convey to the sense of touch stimuli which would normally be addressed to other modalities; the other class simulates the haptic behavior of other structures whose vibrational properties have previously been characterized.
The field of sensory substitution has been thoroughly investigated since the beginning of the last century. In 1930, von Békésy started investigating the physiology behind tactile perception by drawing a parallel between the tactile and the auditory channels in terms of the mechanism governing the two perception mechanisms . A thorough review of sensory substitution applications can be found in Visell . In a musical context, several interfaces have been produced with the aim of translating sound into perceivable vibrations delivered via vibrotactile displays. Crossmodal mapping techniques can be utilized to perform the translation, identifying sound descriptors to be mapped to properties of vibrotactile feedback.
Karam et al.  developed a general-purpose interface in the form of an augmented chair (the Emoti-Chair) embedded with an array of eight speakers disposed along the back. The authors’ aim was to create a display for deaf people to enjoy music through vibrations. They developed the Model Human Cochlea —a sensory substitution model of the cochlear critical band filter on the back—and mapped different frequency bands of a musical track, rescaled to fit into the frequency range of sensitivity of the skin (see Sect. 4.2), to each of the speakers on the chair. In a related study, Egloff et al.  investigated people’s ability to differentiate between musical intervals delivered via the haptic channel, finding that on the average smallest perceptible difference was a major second (i.e., two semitones). It was also noted that results vary widely due to the sensitivity levels of different receptive fields across the human body. Thus, care must be taken when designing vibrotactile interfaces intended to be used as a means for sensory substitution.
Merchel et al.  developed a prototype mixer equipped with a tactile translation system to be used by sound recording technicians. A mixer augmented with an actuator would allow the user to recognize the instrument playing in the selected track only by means of tactile stimulation: A tactile preview mode would be enabled on the mixer, performing a real-time translation of the incoming audio. Preliminary results show that users were able to recognize different instruments only via the sense of touch; better performance was obtained for instruments producing very low-frequency vibrations (bass) or strong rhythmical patterns (drums). A similar touch screen-based system and related test applications are described in Chap. 12.
In tactile stimulation applications, the vibrational behavior of a vibrating structure is characterized and modeled so as to be able to reproduce it in another interface. Examples in this category include physical modeling of the vibrating behavior of a musical instrument, displayed by means of actuators.
A DMI featuring tactile stimulation capability is the Viblotar by Marshall . The instrument is composed of a long, narrow wooden box equipped with sensors and embedded speakers. Sound is generated from a hybrid physical model of an electric guitar and a flute programmed in the Max/MSP environment. During performance, the instrument rests on the performer’s lap or on a stand. One hand manipulates a long linear position sensor and matching force sensitive resistor (FSR) underneath to “pluck” a virtual string. The location, force, and speed of the motion are mapped to frequency, amplitude, and timbre parameters of the physical model. The other hand operates two small FSRs which control pitch bend up and down. The sound output from the Viblotar can be redirected to external speakers, hence allowing the embedded speakers to function primarily for generating vibrotactile feedback instead of sound output. In this configuration, the sound output is split, with one signal sent directly to the external speakers and another routed through a signal processing module that can produce a variety of customized vibrotactile effects such as compensating for frequency response of loudspeakers, simulating the frequency response of another instrument or amplifying the frequency band to which the skin is most sensitive .
Tactile languages are an attempt to create compositional languages solely addressed to the sense of touch, in which tactile effects are not just simple notifications, issued from the interaction with a system, but can be units or icons for abstract communication mediated by the skin.
An early example of tactile language is the “vibratese,” proposed by Geldard , who aimed at creating a complete new form of tactile communication delivered by voice coil actuators (see Sect. 13.2). Parameters for defining building blocks for the language would be elements such as frequency, intensity, and waveform. A total of 45 unit blocks representing numbers and letters of the English alphabet were produced, allowing for expert users to read at rates up to 60 words per minute.
More recently, much research on tactile languages has been directed toward the development of tactile icons. Brewster and Brown  introduced the notion of tactons, i.e., tactile icons to be used to convey non-visual information by means of abstract or meaningful associations, which have been used to convey information about interaction with mobile phones . Enriquez and MacLean  studied the learnability of tactile icons delivered to the fingertips by means of voice coil-like actuators. By modulating frequency, amplitude and rhythm of the vibration, they produced a set of 20 icons, which were tested in a user-based study organized in two sessions, two-weeks apart. Participants recognition rates reached 80% in the first session after 10 min of familiarization with the system and more than 90% during the second session.
In a musical context, attempts to create compositional languages for the sense of touch can be found in the literature. Gunther  developed the Skinscape system, a tactile compositional language whose building blocks varied in frequency, intensity, envelope, spectral content of vibrations, and spatial position on the body of the user. The language was at the base of the Cutaneous Grooves project by Gunther and O’Modhrain , in which it was used to compose a musical piece to be accompanied by vibrations delivered by a custom-built set of suits embedded with various kinds of actuators.
In terms of tactons, we are not aware of any study evaluating their effectiveness in the context of music performance and practice. This is the object of the remainder of this chapter, where we present the design and evaluation of tactile icons for expert musicians.
10.3 Development and Evaluation of Tactile Icons for Music Performance
Our focus in this section will be on the development of a tactile language and its application in designing a language of vibrotactile cues to be used by musicians. We present the design process behind the tactons we developed, and present a methodology for evaluating their effectiveness when delivered via tactile-augmented garments. Our work was conducted in the context of Musicking the Body Electric, a four-year (2014–2018) multidisciplinary project involving researchers from the fields of haptics, music technology, music education, composition, and wearable electronics.3
The ultimate goal of the project is to develop tactile-augmented suits and a language of tactons  to be used as building blocks for a wearable score system. The language will allow composers to convey musical information via tactile stimulation in the context of a music performance in which musicians are free to walk in the performance space. The augmented garments will be able to sense the location of the musicians in the performance space and also the position of musicians relative to one another. This, for instance, would allow each of the suits to be aware of the proximity of other musicians in the room and cue them to play a given section of the piece by delivering the corresponding tactile icon.
10.3.1 Hardware and Software
The work we present is the result of the first tests conducted on two specialized garments produced for the project: an augmented belt embedded with six vibrating actuators and an elastic band embedded with a single actuator that could be worn around an arm or leg. These garments were developed taking advantage of the hardware and software we contributed to create for Ilinx, a multisensory art installation featuring a whole-body suit embedded with vibrating actuators .
The garments created for Ilinx feature a custom-designed Arduino-compatible board embedded with motor drivers and a Serial Peripheral Interface (SPI) bus. Each board can control up to six actuators independently and is connected to a BeagleBone Black (BBB)4 minicomputer via an Ethernet to SPI adapter. The BBB implements an Open Sound Control (OSC) parser which receives control commands from a Max-based synthesizer via a wireless network, and dispatches the message to the correct board and actuator via SPI.
Solarbotics VPM25 actuators were used for the garments. This ERM type (see Sect. 13.2) of actuator was chosen for its ready availability, low cost, and simple design and had previously been characterized for both their physical and perceptual properties .
A second garment was also introduced, consisting of a single actuator mounted on an adjustable band made of stretchable fabric, which could be easily worn on body parts such as wrist, upper arm, or ankle.
10.3.2 Symbolic and Musical Tactons: Design and Evaluation
In the early phase of the project, our approach consisted in designing two sets of tactons, to be reproduced, respectively, by the belt and the band. The former would be used to convey symbolic tactons, i.e., abstract patterns that musicians would need to learn and associate with specific musical elements, for instance sections of a score, chords. The latter would deliver instead musical tactons, i.e., tactons which carry a unique musical meaning, attached to the temporal properties of the tacton itself.
10.3.2.1 Symbolic Tactons
A spatial dimension, associated with the definition of geometrical patterns on the hexagon schematizing the disposition of the six actuators around the waist (see Fig. 10.3);
A global temporal dimension. Once the geometrical pattern of the tacton has been defined, the temporal order or sequence in which the actuators are activated can shape the global perception of the tactile effect;
An individual temporal dimension, which pertains to the properties of the envelope of the vibrotactile signal for each individual actuator.
For the design of the symbolic tactons, we applied a heuristic approach: We defined several geometric patterns which we hypothesized would feature unique characteristics, making them easily distinguishable from one another; we then implemented these patterns, together with preliminary global and individual temporal properties, on a Max-based tactile sequencer we programmed to control the belt; a music pedagogy doctoral researcher (Audrey-Kristel Barbeau) would then test the icons and provide immediate feedback to allow us to proceed to another iteration of the design process.
The symbolic tactons we designed for the belt do not carry any musical or other meaning per se, and need to be learned by the performers to be proficiently used to convey musical information. These icons can be mapped to several musical functions, such as chords or sections of a piece, and these mappings also need to be mastered by musicians to be correctly interpreted.
10.3.2.2 Musical Tactons
While the symbolic tactons were designed by first creating geometric and temporal patterns for the vibrotactile stimuli which could later be mapped arbitrarily to musical functions, design of musical tactons for the single-actuator band took the opposite approach. For these, we started by determining the set of musical information this actuator would deliver. From experiences we gathered in our previous work , we hypothesized that a single-actuator configuration could be used to provide tempo cues, as well as information about articulation and dynamics.
10.3.2.3 Preliminary Evaluation
We conducted a preliminary evaluation of both symbolic and musical tactons’ design with our two musicians, who performed a series of musical tasks we associated with each of the icons. It was important for us to evaluate the learnability and recognition rate of the tactons in the context of music performance in order to establish if musicians actively engaged in a musical task could reliably recognize and respond to the given tactile icons.
Two repetitions of task 1 were performed 10 min apart. The results are depicted in Fig. 10.7a and show the average recognition rate of twenty randomly ordered tactons for each of the two repetitions. For the first trial, the two musicians correctly identified 86 and 77% of the tactons, respectively. In the second repetition, both performers achieved 88%.
A second session took place two weeks after session 1, testing both symbolic and musical tactons. Following the previously described protocol, we performed task 1 first, whose results are depicted in Fig. 10.8a.
10.3.2.4 Musician’s Feedback and Discussion
The two testing sessions with the undergraduate musicians show several patterns: Performers’ recognition rate in both sessions was consistently over 80% for task 1, even after only 20 min of practice with the belt (consistent with findings in Enriquez and MacLean ). This suggests that for both the musical and the symbolic tactons, we were able to design learnable and distinguishable tactile icons.
When looking at the data for task 2, in both sessions we can observe important differences between the two performers. Performer 1 consistently achieved better results than performer 2, who afterward reported that the task could become quickly overwhelming, especially in the second session. This suggests that the complexity of the task prevented performer 2 from simultaneously paying attention to both types of tactile icons while reading and playing the melodies on the instrument. Performer 2’s performance nonetheless improved over time, as visible in Fig. 10.8b, going from a 25% recognition rate for the musical icons in trial one to almost 80% in trial three.
Participant 1 scored above 80% in most of the tasks across the two sessions, and two trends can be identified: For both sessions, performer 1’s performance in the musical task decreased in trial three, compared to the first two trials. This might be due to the presence of adaptation effects which would decrease the sensitivity to the tactile icons. The musician stated that the tasks were not too demanding and that the icon design allowed to easily differentiate the tactile effects.
Overall, the variation between the two participants could be caused by different levels of proficiency on their instrument and ability to sight-read, despite their similar self-assessed musical expertise: Participant 1 was very confident in the sight-reading and performance of the melodies we proposed, while for participant 2 this task proved to be quite demanding, as demonstrated by the frequent hesitation in performing the given melodies which can be heard in the audio recording of the testing sessions. The different postures adopted by the two musicians when playing the saxophone and the guitar, respectively, could also be partly responsible for the variation between the two participants, but this aspect would require an investigation conducted on a larger group of musicians. Additionally, the limited number of repetitions and subjects makes it difficult to draw definitive conclusions about significant trends over repetitions, as randomness may have had an impact on the results.
The observations reported above indicate that a satisfying degree of tactile icon recognition can be reached for both musical and symbolic tactons during the performance of a musical task, provided a high degree of confidence and expertise on the performer’s side. While all the musical tactons were equally well recognized during the two testing session, symbolic tactile icons 5 and 6 were the most problematic ones in terms of recognition rates. Tacton 5 would often be confused with tacton 9 since, as reported by performer 1, the vibration coming from the two actuators on the sides would sometimes go unnoticed. This could be due to lower skin sensitivity in the waist area, which, combined with its peculiar geometrical pattern, made tacton 6 also difficult to recognize at times.
Ultimately, our results confirm that the transparency of a tacton  is not an absolute property of the tactile icon itself, but is very much influenced by the global context in which tactile information is being transmitted to users and to their available cognitive resources .
10.3.3 Implementation into Live Performance
Following the evaluation sessions, the wearable score system was put into practice with a performance of 40 Icons about Art/Music composed by Sandeep Bhagwati and performed by trombonist Felix Del Tredici.6 The piece was the first étude to be composed for the augmented belt  and consisted of ten random repetitions of four musical tasks, each associated with one of the four symbolic icons chosen from the ten described in Sect. 10.3.2.1. In rehearsals, we worked with the performer to identify the set of four tactons to be used for the piece, which led to the selection of tactons 2, 3, 4, and 6 in Fig. 10.3. During the performance, a tacton would be delivered to the performer via the belt. He then had to execute the associated task once the corresponding tactile icon was recognized.
Following the performance, we asked the performer about his experience during the piece. He found the four icons easy to recognize, while admitting that it took a considerable effort to pay attention to the vibrations coming from the belt while performing the musical tasks.
In this chapter, we presented a literature review of the use of haptic technology in music performance. Our focus was the design and implementation of solutions incorporating active vibrotactile feedback and stimulation. We presented a threefold taxonomy of applications in this domain and provided examples for each one of the categories we defined: tactile notification, translation, and languages.
In the second part of the chapter, we focused on tactile languages and presented the results achieved in Musicking the body electric, a multidisciplinary project in which we contributed by designing and evaluating the use of tactile icons to convey score information to expert musicians. Several researchers have evaluated the use of such icons. To our knowledge, no previous evaluation of the use of this type of tactile communications has been performed in the context of musical interaction. For our purposes, it was important to evaluate our approach in the performance of authentic musical tasks. The evaluation we presented shows that our design paradigms for the tactile icons allow for recognition rate consistently around 80% after 20 min of familiarization with the system. The musical tasks we proposed, on the other hand, seem to impact these recognition rates in a way that is dependent on the users’ musical expertise, and the effect of learning is visible already during a single session.
Work continues on Musicking the body electric in all areas. Bhagwati composed Fragile Disequilibria , a piece for solo trombone and four spectators, for which new suit prototypes were designed with multiple ERM motors placed along the arms and legs, across the back and around waist. New materials and technologies are also being tested to design a more robust and flexible platform for haptic garments that can be adapted to a number of different performance contexts. In addition to prototypes developed specifically for this project, a new modular wireless tactile system has also been introduced, where an array of self contained, single-actuator devices called Vibropixels can be placed flexibly on a garment, allowing them to be moved or reconfigured depending on the application [24, 25]. Finally, new compositions are being created for the suits to explore some of the novel possibilities afforded by a vibrotactile score system, most notably the expanded use of physical space and movement among performers.
We refer here to the case of general-purpose interfaces developed for musical applications. These displays are generally conceived as portable/wearable devices to be used by musicians either practicing or performing on stage. Kinaesthetic devices, on the other hand, are generally much larger in scale and are hence difficult to integrate into the design of a portable, general-purpose musical interface.
Stimuli whose intensity exceeds vibrotactile thresholds and are thus perceivable (see Sect. 4.2).
Principal investigators: Sandeep Bhagwati (Matralab, Concodia University, Montreal), Marcelo M. Wanderley (McGill University, Montreal), Isabelle Cossette (MPBL, McGill Univesrity), Joanna Berzowska (XS Labs, Concordia University); funded by the Social Sciences and Humanities Research Council of Canada.
https://beagleboard.org/black (last accessed on December 17, 2017).
https://solarbotics.com/product/vpm2/ (last accessed on December 17, 2017).
http://www.felixdeltredici.com/ (last accessed on Dec. 17, 2017).
We would like to thank the Social Sciences and Humanities Research Council (SSHRC) of Canada and the Natural Sciences and Engineering Research Council (NSERC) of Canada Discovery grant for supporting this research. Special thanks to Sandeep Bhagwati, Isabelle Cossette, Audrey-Kristel Barbeau, Deborah Egloff, Joanna Berzowska, Alexandra Bachmayer, and all the collaborators to the Musicking the body electric project.
- 3.Bhagwati, S., Cossette, I., Berzowska, J., Wanderley, M., Sullivan, J., Egloff, D., Giordano, M., Basanta, A., Stein, J., Browne, J., Bachmeyer, A., Del Tredici, F., Albu, S., Klein, J.: Musicking the Body Electric: the “Body:Suit:Score” as a Polyvalent Score Interface for Situational Scores. In: Proceedings of the 2nd International Conference on Technologies for Music Notation and Representation (TENOR), Cambridge, UK (2016)Google Scholar
- 4.Birnbaum, D.M., Wanderley, M.M.: A systematic approach to musical vibrotactile feedback. In: Proceedings of the International Computer Music Conference (ICMC), Copenhagen, Denmark (2007)Google Scholar
- 5.Bongers, B.: Physical Interfaces in the Electronic Arts—Interaction Theory and Interfacing Techniques for Real-time Performance. In: Wanderley, M.M., Battier, M. (eds.) Trends in Gestural Controls of Music, pp. 124–168. IRCAM—Centre Pompidou, Paris, France (2000)Google Scholar
- 6.Brewster, S.A., Brown, L.M.: Non-visual information display using tactons. In: Proceedings of the CHI’04 Extended Abstracts on Human Factors in Computing Systems, vol. 28, pp. 787–788. ACM, Vienna, Austria (2004)Google Scholar
- 7.Brown, L.M., Brewster, S.A., Purchase, H.C.: A First Investigation into the Effectiveness of Tactons. In: Proceedings of the 1st Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Pisa, Italy, pp. 167–176 (2005)Google Scholar
- 8.Brown, L.M., Kaaresoja, T.: Feel who’s talking: using tactons for mobile phone alerts. In: Proceedings of the CHI’06 conference on Human factors in Computing Systems, pp. 604–609. ACM Press, New York, NY, USA (2006)Google Scholar
- 10.Cook, P.R., Scavone, G.P.: The synthesis toolkit (STK). In: Proceedings of the International Computer Music Conference (ICMC), Beijing, China (1999)Google Scholar
- 11.de Laubier, S., Goudard, V.: Meta-Instrument 3: a look over 17 years of practice. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), Paris, France, pp. 288–291 (2006)Google Scholar
- 12.Egloff, D., Frissen, I., Wanderley, M.M.: Vibrotactile melodic interval discrimination on the index finger of the non-dominant hand. In: Proceedings of the Eurohaptics conference, London, UK (2016)Google Scholar
- 13.Enriquez, M., MacLean, K.E.: The role of choice in longitudinal recall of meaningful tactile signals. In: IEEE Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Reno, NV, USA, pp. 49–56 (2008)Google Scholar
- 14.Essl, G., O’Modhrain, S.: Scrubber: an interface for friction-induced sounds. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), Vancouver, Canada, pp. 70–75 (2005)Google Scholar
- 15.Frid, E., Giordano, M., Schumacher, M.M., Wanderley, M.M.: Physical and perceptual characterization of a tactile display for a live-electronics notification system. In: Proceedings of the Joint International Computer Music Conference and Sound and Music Computing Conference (ICMC–SMC), Athens, Greece (2014)Google Scholar
- 17.Giordano, M.: Vibrotactile feedback and stimulation in music performance. Ph.D. thesis, McGill University, Canada (2016)Google Scholar
- 18.Giordano, M., Hattwick, I., Franco, I., Egloff, D., Frid, E., Lamontagne, V., TeZ, C., Salter, C., Wanderley, M.M.: Design and implementation of a whole-body haptic suit for “Ilinx”, a multisensory art installation. In: Proceedings of the Sound and Music Computing conference (SMC), Maynooth, Ireland, pp. 169–175 (2015)Google Scholar
- 19.Giordano, M., Wanderley, M.M.: Perceptual and technological issues in the design of vibrotactile-augmented interfaces for music technology and media. In: Oakley, I., Brewster, S. (eds.) Haptic and Audio Interaction Design (HAID). Lecture Notes in Computer Science, vol. 7989, pp. 89–98. Springer, Berlin, Heidelberg (2013)CrossRefGoogle Scholar
- 21.Grosshauser, T., Hermann, T.: Augmented haptics—an interactive feedback system for musicians. In: Altinsoy, M.E., Jekosch, U., Brewster, S. (eds.) Haptic and Audio Interaction Design (HAID). Lecture Notes in Computer Science (LNCS), vol. 5763, pp. 100–108. Springer, Berlin, Heidelberg (2009)CrossRefGoogle Scholar
- 22.Gunther, E.: Skinscape: a tool for composition in the tactile modality. Master’s thesis, Massachusetts Institute of Technology, USA (2001)Google Scholar
- 25.Ignoto, P., Hattwick, I., Wanderley, M.M.: Development of a vibrotactile metronome to assist in conducting contemporary classical music. In: Proceedings of the International Conference on Human Factors in Robots and Unmanned Systems (AHFE), Los Angeles, CA, USA, pp. 248–258 (2018)Google Scholar
- 26.Karam, M., Russo, F.A., Branje, C., Price, E., Fels, D.I.: Towards a model human cochlea: sensory substitution for crossmodal audio-tactile displays. In: Proceedings of the Graphics Interface Conference, pp. 267–274. Canadian Information Processing Society (2008)Google Scholar
- 29.Keele, S.W.: Attention and Human Performance. Goodyear Pub. Co (1973)Google Scholar
- 31.MacLean, K.E.: Designing with haptic feedback. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 783–788 (2000)Google Scholar
- 33.MacLean, K.E., Enriquez, M.: Perceptual design of haptic icons. In: Proceedings of the Eurohaptics Conference, Dublin, Ireland, pp. 351–363 (2003)Google Scholar
- 34.Marshall, M.T.: Physical interface design for digital musical instruments. Ph.D. thesis, McGill University, Canada (2008)Google Scholar
- 35.Marshall, M.T., Wanderley, M.M.: Examining the effects of embedded vibrotactile feedback on the feel of a digital musical instrument. In: Proceedings of the Conference on New Interfaces for Musical Expressian (NIME), Oslo, Norway, pp. 399–404 (2011)Google Scholar
- 36.Merchel, S., Altinsoy, M.E., Stamm, M.: Tactile music instrument recognition for audio mixers. In: Proceedings of the Audio Engineering Society Convention (AES), London, UK (2010)Google Scholar
- 37.Michailidis, T., Berweck, S.: Tactile feedback tool: approaching the foot pedal problem in live electronic music. In: Proceedings of the International Computer Music Conference (ICMC), Huddersfield, UK (2011)Google Scholar
- 38.Michailidis, T., Bullock, J.: Improving performers’ musicality through live interaction with haptic feedback: a case study. In: Proceedings of the Sound and Music Computing Conference (SMC), Padua, Italy (2011)Google Scholar
- 39.Miranda, E.R., Wanderley, M.M.: New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. The Computer Music and Digital Audio Series, A-R Edn. (2006)Google Scholar
- 40.Nichols, C.: The vBow: development of a virtual violin bow haptic human-computer interface. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), pp. 1–4 (2002)Google Scholar
- 41.O’Modhrain, S., Essl, G.: PebbleBox and CrumbleBag: tactile interfaces for granular synthesis. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), pp. 74–79 (2004)Google Scholar
- 42.O’Modhrain, S., Gillespie, R.B.: The moose: a haptic user interface for blind persons. In: Proceedings of the International World Wide Web Conference WWW6, Santa Clara, CA, USA (1997)Google Scholar
- 43.Puckette, M., Settel, Z.: Nonobvious roles for electronics in performance enhancement. In: Proceedings of the International Computer Music Conference (ICMC), Tokyo, Japan, pp. 134–134 (1993)Google Scholar
- 44.Qian, H., Kuber, R., Sears, A.: Tactile notifications for ambulatory users. In: Proceedings of the CHI’13 Extended Abstracts on Human Factors in Computing Systems, pp. 1569–1574. ACM, Paris, France (2013)Google Scholar
- 45.Schumacher, M., Giordano, M., Wanderley, M.M., Ferguson, S.: Vibrotactile notification for live electronics performance: a prototype system. In: Proceedings of the Internationa Symposium on Computer Music Multidisciplinary Research (CMMR), Marseille, France (2013)Google Scholar
- 47.Sinyor, E., Wanderley, M.M.: GyroTyre : A dynamic hand-held computer-music controller based on a spinning wheel. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), Vancouver, Canada, pp. 42–45 (2005)Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.