Advertisement

Design of Vibrotactile Feedback and Stimulation for Music Performance

  • Marcello GiordanoEmail author
  • John Sullivan
  • Marcelo M. Wanderley
Open Access
Chapter
  • 7.6k Downloads
Part of the Springer Series on Touch and Haptic Systems book series (SSTHS)

Abstract

Haptics, and specifically vibrotactile-augmented interfaces, have been the object of much research in the music technology domain: In the last few decades, many musical haptic interfaces have been designed and used to teach, perform, and compose music. The investigation of the design of meaningful ways to convey musical information via the sense of touch is a paramount step toward achieving truly transparent haptic-augmented interfaces for music performance and practice, and in this chapter we present our recent work in this context. We start by defining a model for haptic-augmented interfaces for music, and a taxonomy of vibrotactile feedback and stimulation, which we use to categorize a brief literature review on the topic. We then present the design and evaluation of a haptic language of cues in the form of tactile icons delivered via vibrotactile-equipped wearable garments. This language constitutes the base of a “wearable score” used in music performance and practice. We provide design guidelines for our tactile icons and user-based evaluations to assess their effectiveness in delivering musical information and report on the system’s implementation in a live musical performance.

Keywords

Vibrotactile Feedback Musical Performance Tactile Icons Wear Scores Tactons 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

10.1 Introduction

In recent years, the widespread availability of smartphones and tablet computers made vibrotactile technology—in the form of actuators specifically designed to stimulate a user’s sense of touch via vibration—inexpensive and readily available. Haptic researchers, both in academic and industrial contexts, have been designing ways of communicating via the sense of touch by means of tactile effects used to provide information such as: navigational cues [50], textures [30], or notifications [44]. Systematic studies have been conducted to assess the efficiency of these effects in well-defined contexts, and new prototypes and applications are constantly being investigated.

In the music domain, the sense of touch can be used to convey relevant musical information, such as articulation [43] and timing [51], especially in professional performances [29]. Several haptic interfaces for music performance and practice have been created in the last two decades, but for very few of these a thorough evaluation of their effectiveness has been conducted.

In this chapter, we present our work in the development and preliminary evaluation of meaningful ways to provide information to performers via the sense of touch for music performance and practice. Our research, conducted in the context of a multidisciplinary project involving haptic researchers, composers, and wearable designers, is aimed at the development of a language of tactile icons specifically designed to convey musical information to professional musicians. These icons, delivered via specialized garments equipped with arrays of vibrotactile actuators, have been evaluated to determine their effectiveness and reliability. They will be used as the building blocks of a wearable score language, which composers will use to create new pieces and art installations.

To provide a theoretical framework for this research, we present a brief overview of the current state of haptic feedback and stimulation in music technology. We expand the classical models of digital musical instruments (DMIs) [39] to include general-purpose tactile interfaces, i.e., devices where other sensory feedback may not be present and tactile feedback can be arbitrary mapped to external sources of information. We then present a literature review together with a taxonomy of tactile feedback and stimulation. This categorization is aimed at emphasizing the different functional roles that haptic technology can achieve in conveying musically relevant information.

10.2 Haptic Feedback in Music Technology

Haptic technology has been widely used in the development of interfaces for musical expression and musical interaction, and two main classes of devices can be identified in this context: DMIs and general-purpose haptic interfaces.

In traditional musical instruments, the tactile and kinaesthetic feedback coming from the resonating parts of the instrument give the performer important information about their interaction [1, 20, 28, 43] (see Chap. 2). In DMIs, the decoupling of gesture acquisition from sound synthesis has the important effect of breaking the mechanical feedback loop between performer and sound-producing structures. Haptic feedback becomes then an arbitrary design factor [31], and the choice of actuators and signals used to drive them (see Sect.  13.2) defines the instrument’s architecture.

Haptic devices can provide tactile cues during performance with DMIs, not only if embedded into the instruments themselves, but also when deployed separately by means of tactile displays and wearable devices that can be used to go beyond the direct performer–instrument interaction. In the context of music performance, these devices, which we refer to as general-purpose haptic interfaces, can convey information about performers’ interactions with a live-electronics system [37] or as learning tools to direct and guide users’ gestures via vibrotactile feedback [49] (see also Chap.  11). They can also be used to convey score cues to a performer on stage [45] by means of abstract languages of tactile icons [33]. In this context, the distinction between feedback and stimulation becomes clear: The former is a direct response of the instrument or the general-purpose interface to a user’s action; the latter is not issued from a player–device interaction, but it is a means of communication with the user, mediated by the tactile actuators in the interface, which can be used to convey any sort of information.

These displays usually provide either localized (i.e., single body site) or distributed vibrations (via actuators placed on multiple body sites), requiring the design of tactile effects more centered on temporal or spatial properties, respectively, or a combination of both.

10.2.1 Models of Haptic-Enabled Interfaces

The relationship between performer, haptic-enabled musical interface (either general-purpose device or DMI), and audience can be complex, and a number of abstract models of the interaction between these components can be found in the literature. In the case of DMIs several models exist, each of which emphasizes different aspects of the instrument’s design. Marshall [34] reviews four of these models [4, 5, 9, 54] and proposes a hybrid model merging characteristics across them.

In Fig. 10.1, we present an extension of this model, which is a representation of the interaction with either haptic-enabled DMIs or general-purpose devices. While the former can provide the performer with both kinaesthetic or tactile feedback, the latter are usually implemented as vibrotactile displays, for reasons that are mainly to be found in current technology limitations.1 As mentioned above, the haptic channel does not need to be limited to the display of feedback issued as a direct response to performers’ actions, but can be mapped arbitrarily to convey information from external sources such as environmental variables or score parameters. This is represented by the external information source in our model.
Fig. 10.1

Model of a haptic DMI and general-purpose haptic device. In both devices, a haptic generator is used to produce haptic feedback and stimulation, which is issued from mapping of sensor data or external information. The simultaneous use of both types of devices is also possible, and sensor data from either device could be mapped to the haptic generator of the other

10.2.2 Haptic-Enabled Interfaces

Haptic-enabled interfaces for music performance can be categorized according to the way they deliver haptic feedback and stimulation to the final users. Both DMIs and general-purpose devices can address either the kinaesthetic or the tactile modality, and this can be done in an active or a passive way [5]: Passive feedback and stimulation come from the inherent physical properties of the interface and are not issued by the system’s haptic generator; active interfaces implement a haptic generator to provide user with the designed kinaesthetic and tactile effects.

We will present some of the most important devices present in the literature following these two categories and provide a threefold taxonomy for the active tactile case.

10.2.2.1 Passive Kinaesthetic Feedback

Passive kinaesthetic feedback and stimulation are inherent to the physical characteristics of the controller, and do not require any externally synthesized signal.

O’Modhrain and Essl developed three DMIs that implement passive kinaesthetic feedback. The Pebble Box and the Crumble Bag [41] were used to control an event-based granular synthesizer: the Pebble Box consists of a box filled with different-sized pebble stones and a microphone that picks up the noise produced by the collisions between pebbles. The kinaesthetic feedback offered by the interface comes from the physical properties of the pebbles themselves, and the impact sounds act as triggering events on the granular synthesizer. The Crumble Bag follows the same patter, and it is aimed to take advantage of natural “grabbing gestures.” A fabric bag is filled with different materials that provide haptic feedback, and a small microphone in the bag provides the necessary event triggers to the algorithm. The Scrubber [14] also implemented the same approach: an eraser embedded with a force sensor and two microphones were used to control the synthesis of friction sounds, synthesized by means of granular or wavetable synthesis. The haptic feedback again was directly issued by the manipulation of the device dragged along a surface.

Sinyor and Wanderley [47] developed the Gyroyre, a handheld controller based on a spinning wheel, in which the kinaesthetic feedback comes directly from the dynamic properties of the system. The mapping and synthesis algorithm are designed to take advantage of the haptic feedback, and the interface can be used for different musical applications, sequencing or modifying effects’ parameters.

10.2.2.2 Active Kinaesthetic Interfaces

Active kinaesthetic feedback is the response of the controller to the user’s actions, usually by means of synthesized signals supplied into motors or actuators, which stimulate kinaesthetic receptors. This is most commonly referred to as force feedback.

The earliest example of a force-feedback device specifically developed for musical applications is probably the Transducteur Gestuel Rétroactif (TGR) developed at ACROE, whose development is described in Sect.  8.3. This device was recently used by Sinclair et al. [46] to investigate velocity estimation methods in the haptic rendering of a bowed string.

Another classical example is the Moose, developed by O’Modhrain and Gillespie [42], consisting of a plastic puck that the user can manipulate in a 2D space, which is attached to flexible metal bars, connected to linear motors. Two encoders sense the movements of the puck, and the motors provide the correspondent force feedback. The device was used in a bowing test, using a virtual string programmed in Synthesis ToolKit (STK) [10], where the presence of friction between the bow and the string was simulated using the haptic device.

The vBow by Nichols [40] is a violin-like controller that uses a series of servomotors and encoders to sense the movement of a rod, acting as the bow, connected to a metallic cable. In its last incarnation, the vBow is capable of sensing moment in 4-DoF and producing haptic feedback accordingly.

More recently, Berdahl and Kontogeorgakopoulos [2] developed the FireFader, a motorized faders using sensors and DC motors to introduce musicians to haptic controllers. Both the software and hardware used for the project are open-source, allowing musicians to customize the mapping of the interface to their specific needs. Applications of the device are described in Chap.  9.

10.2.2.3 Passive Tactile Interfaces

Passive tactile is a form of primary feedback, which leverages the use of different types of materials in a controller for musical expression. The properties of these materials (e.g., stiffness, flexibility) can affect the ergonomics of the instrument and its feel in the user’s hands.

As an example, the Meta-Instrument [11] has the form of a partial exoskeleton embedded with buttons that the performer uses to trigger samples and events in the sound; the performer’s gestures are captured via sensors in the arms and mapped to various effects. The buttons embedded in the controller are covered in a layer of foam, providing the user with immediate passive feedback about the level of pressure applied.

10.2.2.4 Active Tactile Feedback and Stimulation: A Taxonomy for Musical Interaction

Active tactile feedback and stimulation are the main focus of this chapter, and for this reason we provide a more in-depth analysis of the related literature, as well as an updated taxonomy, based on Giordano and Wanderley [19], which will help categorize examples in this field.

We propose a classification identifying in active tactile feedback and stimulation three different categories according to the function that the tactile effects have in the interface design: tactile notification, tactile translation, and tactile languages.

Tactile Notification

The most straightforward application of tactile stimulation is intended for notifying the users about events taking place in the surrounding environment or about results of their interaction with a system. The effects designed for this kind of applications can be as simple as single, supra-threshold stimuli2 aimed at directing users’ attention, but they can also be more complex, implementing temporal envelopes and/or spatial patterns.

Michailidis and Berweck [37] and Michailidis and Bullock [38] have explored solutions to provide haptic feedback in live-electronics performance. The authors developed the Tactile Feedback Tool, a general-purpose interface using small vibrating motors embedded in a glove. The interface gave musicians information about the successful triggering of effects in a live-electronics performance, using an augmented trumpet or a foot pedal switch. This device leverages the capacity of the tactile sense to attract users’ attention, while not requiring them to lose focus on other modalities, which would have been the case with the use of onstage visual displays.

Van der Linden et al. [49] implemented a whole-body general-purpose vibrotactile device. The authors used a motion capture system and a suit embedded with vibrating motors distributed over the body to enhance the learning process of bowing for novice violin players. A set of ideal bowing trajectories was computed using the motion capture system; when practicing, the players’ postures would be compared in real time with the predefined ideal trajectories. If the distance between any two corresponding points in the two trajectories exceeded the threshold value, the motor spatially closer to that point would vibrate, notifying the users to correct their posture. The authors conducted a study in which several players used the suit during their violin lessons. Results showed an improved coordination of the bowing arm, and participants reported an enhancement in their body awareness produced by the feedback.

A similar solution was developed by Grosshauser and Hermann [21], which used a vibrating actuator embedded in a violin bow to correct hand posture. Using accelerometers and gyroscopes, the position of the bow could be compared in real time to a given trajectory, and the tactile feedback would automatically activate to notify the users about their wrong posture.

Tactile Notification

With tactile translation, we refer to two separate classes of applications: One class implements sensory substitution techniques to convey to the sense of touch stimuli which would normally be addressed to other modalities; the other class simulates the haptic behavior of other structures whose vibrational properties have previously been characterized.

Sensory Substitution

The field of sensory substitution has been thoroughly investigated since the beginning of the last century. In 1930, von Békésy started investigating the physiology behind tactile perception by drawing a parallel between the tactile and the auditory channels in terms of the mechanism governing the two perception mechanisms [53]. A thorough review of sensory substitution applications can be found in Visell [52]. In a musical context, several interfaces have been produced with the aim of translating sound into perceivable vibrations delivered via vibrotactile displays. Crossmodal mapping techniques can be utilized to perform the translation, identifying sound descriptors to be mapped to properties of vibrotactile feedback.

Karam et al. [27] developed a general-purpose interface in the form of an augmented chair (the Emoti-Chair) embedded with an array of eight speakers disposed along the back. The authors’ aim was to create a display for deaf people to enjoy music through vibrations. They developed the Model Human Cochlea [26]—a sensory substitution model of the cochlear critical band filter on the back—and mapped different frequency bands of a musical track, rescaled to fit into the frequency range of sensitivity of the skin (see Sect.  4.2), to each of the speakers on the chair. In a related study, Egloff et al. [12] investigated people’s ability to differentiate between musical intervals delivered via the haptic channel, finding that on the average smallest perceptible difference was a major second (i.e., two semitones). It was also noted that results vary widely due to the sensitivity levels of different receptive fields across the human body. Thus, care must be taken when designing vibrotactile interfaces intended to be used as a means for sensory substitution.

Merchel et al. [36] developed a prototype mixer equipped with a tactile translation system to be used by sound recording technicians. A mixer augmented with an actuator would allow the user to recognize the instrument playing in the selected track only by means of tactile stimulation: A tactile preview mode would be enabled on the mixer, performing a real-time translation of the incoming audio. Preliminary results show that users were able to recognize different instruments only via the sense of touch; better performance was obtained for instruments producing very low-frequency vibrations (bass) or strong rhythmical patterns (drums). A similar touch screen-based system and related test applications are described in Chap.  12.

Tactile Stimulation

In tactile stimulation applications, the vibrational behavior of a vibrating structure is characterized and modeled so as to be able to reproduce it in another interface. Examples in this category include physical modeling of the vibrating behavior of a musical instrument, displayed by means of actuators.

A DMI featuring tactile stimulation capability is the Viblotar by Marshall [35]. The instrument is composed of a long, narrow wooden box equipped with sensors and embedded speakers. Sound is generated from a hybrid physical model of an electric guitar and a flute programmed in the Max/MSP environment. During performance, the instrument rests on the performer’s lap or on a stand. One hand manipulates a long linear position sensor and matching force sensitive resistor (FSR) underneath to “pluck” a virtual string. The location, force, and speed of the motion are mapped to frequency, amplitude, and timbre parameters of the physical model. The other hand operates two small FSRs which control pitch bend up and down. The sound output from the Viblotar can be redirected to external speakers, hence allowing the embedded speakers to function primarily for generating vibrotactile feedback instead of sound output. In this configuration, the sound output is split, with one signal sent directly to the external speakers and another routed through a signal processing module that can produce a variety of customized vibrotactile effects such as compensating for frequency response of loudspeakers, simulating the frequency response of another instrument or amplifying the frequency band to which the skin is most sensitive [34].

Tactile Languages

Tactile languages are an attempt to create compositional languages solely addressed to the sense of touch, in which tactile effects are not just simple notifications, issued from the interaction with a system, but can be units or icons for abstract communication mediated by the skin.

An early example of tactile language is the “vibratese,” proposed by Geldard [16], who aimed at creating a complete new form of tactile communication delivered by voice coil actuators (see Sect.  13.2). Parameters for defining building blocks for the language would be elements such as frequency, intensity, and waveform. A total of 45 unit blocks representing numbers and letters of the English alphabet were produced, allowing for expert users to read at rates up to 60 words per minute.

More recently, much research on tactile languages has been directed toward the development of tactile icons. Brewster and Brown [6] introduced the notion of tactons, i.e., tactile icons to be used to convey non-visual information by means of abstract or meaningful associations, which have been used to convey information about interaction with mobile phones [8]. Enriquez and MacLean [13] studied the learnability of tactile icons delivered to the fingertips by means of voice coil-like actuators. By modulating frequency, amplitude and rhythm of the vibration, they produced a set of 20 icons, which were tested in a user-based study organized in two sessions, two-weeks apart. Participants recognition rates reached 80% in the first session after 10 min of familiarization with the system and more than 90% during the second session.

In a musical context, attempts to create compositional languages for the sense of touch can be found in the literature. Gunther [22] developed the Skinscape system, a tactile compositional language whose building blocks varied in frequency, intensity, envelope, spectral content of vibrations, and spatial position on the body of the user. The language was at the base of the Cutaneous Grooves project by Gunther and O’Modhrain [23], in which it was used to compose a musical piece to be accompanied by vibrations delivered by a custom-built set of suits embedded with various kinds of actuators.

In terms of tactons, we are not aware of any study evaluating their effectiveness in the context of music performance and practice. This is the object of the remainder of this chapter, where we present the design and evaluation of tactile icons for expert musicians.

10.3 Development and Evaluation of Tactile Icons for Music Performance

Our focus in this section will be on the development of a tactile language and its application in designing a language of vibrotactile cues to be used by musicians. We present the design process behind the tactons we developed, and present a methodology for evaluating their effectiveness when delivered via tactile-augmented garments. Our work was conducted in the context of Musicking the Body Electric, a four-year (2014–2018) multidisciplinary project involving researchers from the fields of haptics, music technology, music education, composition, and wearable electronics.3

The ultimate goal of the project is to develop tactile-augmented suits and a language of tactons [7] to be used as building blocks for a wearable score system. The language will allow composers to convey musical information via tactile stimulation in the context of a music performance in which musicians are free to walk in the performance space. The augmented garments will be able to sense the location of the musicians in the performance space and also the position of musicians relative to one another. This, for instance, would allow each of the suits to be aware of the proximity of other musicians in the room and cue them to play a given section of the piece by delivering the corresponding tactile icon.

10.3.1 Hardware and Software

The work we present is the result of the first tests conducted on two specialized garments produced for the project: an augmented belt embedded with six vibrating actuators and an elastic band embedded with a single actuator that could be worn around an arm or leg. These garments were developed taking advantage of the hardware and software we contributed to create for Ilinx, a multisensory art installation featuring a whole-body suit embedded with vibrating actuators [18].

The garments created for Ilinx feature a custom-designed Arduino-compatible board embedded with motor drivers and a Serial Peripheral Interface (SPI) bus. Each board can control up to six actuators independently and is connected to a BeagleBone Black (BBB)4 minicomputer via an Ethernet to SPI adapter. The BBB implements an Open Sound Control (OSC) parser which receives control commands from a Max-based synthesizer via a wireless network, and dispatches the message to the correct board and actuator via SPI.

Solarbotics VPM25 actuators were used for the garments. This ERM type (see Sect.  13.2) of actuator was chosen for its ready availability, low cost, and simple design and had previously been characterized for both their physical and perceptual properties [15].

The wearable designers involved in the project (Joanna Berzowska and Alex Bachmayer, XS Labs, Concordia University) produced the first specialized garment for us to test: a tactile-augmented belt with six equally spaced ERM actuators (Fig. 10.2). The choice of a belt as the first garment to be designed was guided by several reasons: The placement of the actuators on a circle around the user’s waist allowed for more flexibility in terms of tactile effects design; more practically, a belt provides an easier fit compared to leggings or sleeves, for instance [48, 50].
Fig. 10.2

Augmented belt embedded with six vibrating actuators (garment design and manufacturing by J. Berzowska and A. Bachmayer—XS Labs, Concordia University)

A second garment was also introduced, consisting of a single actuator mounted on an adjustable band made of stretchable fabric, which could be easily worn on body parts such as wrist, upper arm, or ankle.

10.3.2 Symbolic and Musical Tactons: Design and Evaluation

In the early phase of the project, our approach consisted in designing two sets of tactons, to be reproduced, respectively, by the belt and the band. The former would be used to convey symbolic tactons, i.e., abstract patterns that musicians would need to learn and associate with specific musical elements, for instance sections of a score, chords. The latter would deliver instead musical tactons, i.e., tactons which carry a unique musical meaning, attached to the temporal properties of the tacton itself.

10.3.2.1 Symbolic Tactons

We identified three different dimensions defining the tacton design space associated with the six-actuator belt:
  • A spatial dimension, associated with the definition of geometrical patterns on the hexagon schematizing the disposition of the six actuators around the waist (see Fig. 10.3);

  • A global temporal dimension. Once the geometrical pattern of the tacton has been defined, the temporal order or sequence in which the actuators are activated can shape the global perception of the tactile effect;

  • An individual temporal dimension, which pertains to the properties of the envelope of the vibrotactile signal for each individual actuator.

Fig. 10.3

Final set of ten symbolic icons developed for the belt (diagram courtesy of A.-K. Barbeau). Each black dot represents one actuator. The hexagon shapes represent the actuators disposed around a user’s waist, with the top two actuators corresponding to the person’s front. Icons 1–4 feature a sequence of actuations which follow the direction indicated by the arrows. For icons 5–10, connected dots represent simultaneous activation of the corresponding actuators, with solid lines happening first, followed by dashed and then dotted lines. Each actuation lasts 200 ms, as per haptic envelope definition, and for each icon the pattern is repeated twice with a 300 ms interval between repetitions

For the design of the symbolic tactons, we applied a heuristic approach: We defined several geometric patterns which we hypothesized would feature unique characteristics, making them easily distinguishable from one another; we then implemented these patterns, together with preliminary global and individual temporal properties, on a Max-based tactile sequencer we programmed to control the belt; a music pedagogy doctoral researcher (Audrey-Kristel Barbeau) would then test the icons and provide immediate feedback to allow us to proceed to another iteration of the design process.

This process lasted over several weeks, after which we finalized a set of ten tactons, depicted in Fig. 10.3. Each of the tactile icons consists of two repetitions of the same pattern which are separated by a fixed time interval. The tactons have a total duration which varies from 1.5 to 2.7 s. For the individual temporal properties, we chose a fixed envelope for all the actuations which features 50 ms of attack, 150 ms of sustain at maximum intensity, and no release time (see Fig. 10.4). We decided to keep the vibrotactile envelope parameters fixed for this initial phase of the project to facilitate the tactons’ learning phase. These tactile icons were proposed to undergraduate music students—a saxophone player (performer 1) and a guitar player (performer 2)—who were the participants for the ensuing evaluation sessions.
Fig. 10.4

Haptic envelopes of each individual actuation composing the icons: 50 ms attack time to 100% duty cycle, 150 ms sustain, and no release time

The symbolic tactons we designed for the belt do not carry any musical or other meaning per se, and need to be learned by the performers to be proficiently used to convey musical information. These icons can be mapped to several musical functions, such as chords or sections of a piece, and these mappings also need to be mastered by musicians to be correctly interpreted.

10.3.2.2 Musical Tactons

While the symbolic tactons were designed by first creating geometric and temporal patterns for the vibrotactile stimuli which could later be mapped arbitrarily to musical functions, design of musical tactons for the single-actuator band took the opposite approach. For these, we started by determining the set of musical information this actuator would deliver. From experiences we gathered in our previous work [15], we hypothesized that a single-actuator configuration could be used to provide tempo cues, as well as information about articulation and dynamics.

Using the heuristic approach based on iterative feedback from A.-K. Barbeau, we designed a set of four musical tactons associated with crescendo, decrescendo, staccato, and legato, respectively, which are shown in Fig. 10.5. These tactons contained a musical meaning attached to the temporal properties of the tacton itself and would ideally require a minimal effort to be correctly interpreted.
Fig. 10.5

Schematization of the envelopes of the four musical tactons developed for the single-actuator band

10.3.2.3 Preliminary Evaluation

We conducted a preliminary evaluation of both symbolic and musical tactons’ design with our two musicians, who performed a series of musical tasks we associated with each of the icons. It was important for us to evaluate the learnability and recognition rate of the tactons in the context of music performance in order to establish if musicians actively engaged in a musical task could reliably recognize and respond to the given tactile icons.

We performed two testing sessions, two weeks apart, following a methodology similar to the one reported in [13]. The musicians had 20 min per session to familiarize themselves with the tactons. Subsequently, they were asked to perform two recognition tasks. In task 1, they experienced a series of tactons and verbally reported the name or number of the tacton they thought they had perceived. In task 2, the musicians were given a score, shown in Fig. 10.6, and asked to perform the melody associated with the perceived icons. The melodies were composed to be easy to sight-read and perform. In the first session only symbolic tactons were tested, while in the second session we tested both symbolic and musical tactons. Performances were audio-recorded and subsequently analyzed to determine recognition rates of the tactile icons in both sessions.
Fig. 10.6

Set of 10 simple melodies, composed by A.-K. Barbeau and associated with the ten symbolic tactile icons. The performer would feel one of the tactons on the augmented belt and perform the corresponding melody

Session 1

Two repetitions of task 1 were performed 10 min apart. The results are depicted in Fig. 10.7a and show the average recognition rate of twenty randomly ordered tactons for each of the two repetitions. For the first trial, the two musicians correctly identified 86 and 77% of the tactons, respectively. In the second repetition, both performers achieved 88%.

For task 2, we provided the musicians with the score shown in Fig. 10.6. This time we asked them to play the melody corresponding to the perceived tactile icon. The musicians were free to play at the tempo they desired. Fifteen randomly ordered icons were tested, and a new icon would be delivered via the belt while the musician was playing the half note ending the previous melody. Task 2 was repeated three times, 10 min apart, and the results are depicted in Fig. 10.7b. The performers reached, respectively, a 92 and 79% recognition rate for the first trial, 92 and 86% for the second trial, and 88 and 71% for the last trial. It is notable that the results declined for both performers in the third trial, factors for which we discuss in Sect. 10.3.2.4.
Fig. 10.7

Recognition rates for session 1 for both task 1 and task 2. Recognition rate is consistently around 80% for both performers

Session 2

A second session took place two weeks after session 1, testing both symbolic and musical tactons. Following the previously described protocol, we performed task 1 first, whose results are depicted in Fig. 10.8a.

For task 2, the musicians wore the belt and the single-actuator elastic band on their left upper arm. A symbolic icon would be delivered via the belt, followed by a musical icon from the single actuator. The musicians were asked to play the corresponding melody following either the articulation or the dynamics indicated by the musical tacton. Results are shown in Fig. 10.8b. For the symbolic icons, the first performer reached a recognition rate of 87% in the first trial, 86% in the second, and 70 and 78% in the third and fourth, respectively. A similar trend can be observed for the musical icons, with a 100% recognition rate in the first repetition, followed by 92, 82, and 88% in the last three trials. The second musician performed less well in this task, reaching a 78% recognition rate for symbolic tactile icons in trial one, 71% for trial two, and 76 and 77% for trials three and four, respectively. For the musical tactons, only 25% of the tactile icons were correctly recognized in trial one, 66% in trial two, and 77 and 57% in trials three and four, respectively.
Fig. 10.8

Recognition rates for session 2 for both task 1 and task 2. Both symbolic and musical tactons were tested in this session. Results show recognition rates consistently around 80% for participant 1, while participant 2 performed less well in task 2

10.3.2.4 Musician’s Feedback and Discussion

The two testing sessions with the undergraduate musicians show several patterns: Performers’ recognition rate in both sessions was consistently over 80% for task 1, even after only 20 min of practice with the belt (consistent with findings in Enriquez and MacLean [13]). This suggests that for both the musical and the symbolic tactons, we were able to design learnable and distinguishable tactile icons.

When looking at the data for task 2, in both sessions we can observe important differences between the two performers. Performer 1 consistently achieved better results than performer 2, who afterward reported that the task could become quickly overwhelming, especially in the second session. This suggests that the complexity of the task prevented performer 2 from simultaneously paying attention to both types of tactile icons while reading and playing the melodies on the instrument. Performer 2’s performance nonetheless improved over time, as visible in Fig. 10.8b, going from a 25% recognition rate for the musical icons in trial one to almost 80% in trial three.

Participant 1 scored above 80% in most of the tasks across the two sessions, and two trends can be identified: For both sessions, performer 1’s performance in the musical task decreased in trial three, compared to the first two trials. This might be due to the presence of adaptation effects which would decrease the sensitivity to the tactile icons. The musician stated that the tasks were not too demanding and that the icon design allowed to easily differentiate the tactile effects.

Overall, the variation between the two participants could be caused by different levels of proficiency on their instrument and ability to sight-read, despite their similar self-assessed musical expertise: Participant 1 was very confident in the sight-reading and performance of the melodies we proposed, while for participant 2 this task proved to be quite demanding, as demonstrated by the frequent hesitation in performing the given melodies which can be heard in the audio recording of the testing sessions. The different postures adopted by the two musicians when playing the saxophone and the guitar, respectively, could also be partly responsible for the variation between the two participants, but this aspect would require an investigation conducted on a larger group of musicians. Additionally, the limited number of repetitions and subjects makes it difficult to draw definitive conclusions about significant trends over repetitions, as randomness may have had an impact on the results.

The observations reported above indicate that a satisfying degree of tactile icon recognition can be reached for both musical and symbolic tactons during the performance of a musical task, provided a high degree of confidence and expertise on the performer’s side. While all the musical tactons were equally well recognized during the two testing session, symbolic tactile icons 5 and 6 were the most problematic ones in terms of recognition rates. Tacton 5 would often be confused with tacton 9 since, as reported by performer 1, the vibration coming from the two actuators on the sides would sometimes go unnoticed. This could be due to lower skin sensitivity in the waist area, which, combined with its peculiar geometrical pattern, made tacton 6 also difficult to recognize at times.

Ultimately, our results confirm that the transparency of a tacton [32] is not an absolute property of the tactile icon itself, but is very much influenced by the global context in which tactile information is being transmitted to users and to their available cognitive resources [44].

10.3.3 Implementation into Live Performance

Following the evaluation sessions, the wearable score system was put into practice with a performance of 40 Icons about Art/Music composed by Sandeep Bhagwati and performed by trombonist Felix Del Tredici.6 The piece was the first étude to be composed for the augmented belt [17] and consisted of ten random repetitions of four musical tasks, each associated with one of the four symbolic icons chosen from the ten described in Sect. 10.3.2.1. In rehearsals, we worked with the performer to identify the set of four tactons to be used for the piece, which led to the selection of tactons 2, 3, 4, and 6 in Fig. 10.3. During the performance, a tacton would be delivered to the performer via the belt. He then had to execute the associated task once the corresponding tactile icon was recognized.

Following the performance, we asked the performer about his experience during the piece. He found the four icons easy to recognize, while admitting that it took a considerable effort to pay attention to the vibrations coming from the belt while performing the musical tasks.

10.4 Conclusions

In this chapter, we presented a literature review of the use of haptic technology in music performance. Our focus was the design and implementation of solutions incorporating active vibrotactile feedback and stimulation. We presented a threefold taxonomy of applications in this domain and provided examples for each one of the categories we defined: tactile notification, translation, and languages.

In the second part of the chapter, we focused on tactile languages and presented the results achieved in Musicking the body electric, a multidisciplinary project in which we contributed by designing and evaluating the use of tactile icons to convey score information to expert musicians. Several researchers have evaluated the use of such icons. To our knowledge, no previous evaluation of the use of this type of tactile communications has been performed in the context of musical interaction. For our purposes, it was important to evaluate our approach in the performance of authentic musical tasks. The evaluation we presented shows that our design paradigms for the tactile icons allow for recognition rate consistently around 80% after 20 min of familiarization with the system. The musical tasks we proposed, on the other hand, seem to impact these recognition rates in a way that is dependent on the users’ musical expertise, and the effect of learning is visible already during a single session.

Work continues on Musicking the body electric in all areas. Bhagwati composed Fragile Disequilibria [3], a piece for solo trombone and four spectators, for which new suit prototypes were designed with multiple ERM motors placed along the arms and legs, across the back and around waist. New materials and technologies are also being tested to design a more robust and flexible platform for haptic garments that can be adapted to a number of different performance contexts. In addition to prototypes developed specifically for this project, a new modular wireless tactile system has also been introduced, where an array of self contained, single-actuator devices called Vibropixels can be placed flexibly on a garment, allowing them to be moved or reconfigured depending on the application [24, 25]. Finally, new compositions are being created for the suits to explore some of the novel possibilities afforded by a vibrotactile score system, most notably the expanded use of physical space and movement among performers.

Footnotes

  1. 1.

    We refer here to the case of general-purpose interfaces developed for musical applications. These displays are generally conceived as portable/wearable devices to be used by musicians either practicing or performing on stage. Kinaesthetic devices, on the other hand, are generally much larger in scale and are hence difficult to integrate into the design of a portable, general-purpose musical interface.

  2. 2.

    Stimuli whose intensity exceeds vibrotactile thresholds and are thus perceivable (see Sect.  4.2).

  3. 3.

    Principal investigators: Sandeep Bhagwati (Matralab, Concodia University, Montreal), Marcelo M. Wanderley (McGill University, Montreal), Isabelle Cossette (MPBL, McGill Univesrity), Joanna Berzowska (XS Labs, Concordia University); funded by the Social Sciences and Humanities Research Council of Canada.

  4. 4.

    https://beagleboard.org/black (last accessed on December 17, 2017).

  5. 5.

    https://solarbotics.com/product/vpm2/ (last accessed on December 17, 2017).

  6. 6.

    http://www.felixdeltredici.com/ (last accessed on Dec. 17, 2017).

Notes

Acknowledgements

We would like to thank the Social Sciences and Humanities Research Council (SSHRC) of Canada and the Natural Sciences and Engineering Research Council (NSERC) of Canada Discovery grant for supporting this research. Special thanks to Sandeep Bhagwati, Isabelle Cossette, Audrey-Kristel Barbeau, Deborah Egloff, Joanna Berzowska, Alexandra Bachmayer, and all the collaborators to the Musicking the body electric project.

References

  1. 1.
    Askenfelt, A., Jansson, E.V.: On vibration sensation and finger touch in stringed instrument playing. Music Percept. 3(9), 311–350 (1992)CrossRefGoogle Scholar
  2. 2.
    Berdahl, E., Kontogeorgakopoulos, A.: The FireFader: simple, open-source, and reconfigurable haptic force feedback for musicians. Comput. Music J. 37(1), 23–34 (2013)CrossRefGoogle Scholar
  3. 3.
    Bhagwati, S., Cossette, I., Berzowska, J., Wanderley, M., Sullivan, J., Egloff, D., Giordano, M., Basanta, A., Stein, J., Browne, J., Bachmeyer, A., Del Tredici, F., Albu, S., Klein, J.: Musicking the Body Electric: the “Body:Suit:Score” as a Polyvalent Score Interface for Situational Scores. In: Proceedings of the 2nd International Conference on Technologies for Music Notation and Representation (TENOR), Cambridge, UK (2016)Google Scholar
  4. 4.
    Birnbaum, D.M., Wanderley, M.M.: A systematic approach to musical vibrotactile feedback. In: Proceedings of the International Computer Music Conference (ICMC), Copenhagen, Denmark (2007)Google Scholar
  5. 5.
    Bongers, B.: Physical Interfaces in the Electronic Arts—Interaction Theory and Interfacing Techniques for Real-time Performance. In: Wanderley, M.M., Battier, M. (eds.) Trends in Gestural Controls of Music, pp. 124–168. IRCAM—Centre Pompidou, Paris, France (2000)Google Scholar
  6. 6.
    Brewster, S.A., Brown, L.M.: Non-visual information display using tactons. In: Proceedings of the CHI’04 Extended Abstracts on Human Factors in Computing Systems, vol. 28, pp. 787–788. ACM, Vienna, Austria (2004)Google Scholar
  7. 7.
    Brown, L.M., Brewster, S.A., Purchase, H.C.: A First Investigation into the Effectiveness of Tactons. In: Proceedings of the 1st Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Pisa, Italy, pp. 167–176 (2005)Google Scholar
  8. 8.
    Brown, L.M., Kaaresoja, T.: Feel who’s talking: using tactons for mobile phone alerts. In: Proceedings of the CHI’06 conference on Human factors in Computing Systems, pp. 604–609. ACM Press, New York, NY, USA (2006)Google Scholar
  9. 9.
    Cook, P.R.: Remutualizing the musical instrument: co-design of synthesis algorithms and controllers. J. New Music Res. 33(3), 315–320 (2004)CrossRefGoogle Scholar
  10. 10.
    Cook, P.R., Scavone, G.P.: The synthesis toolkit (STK). In: Proceedings of the International Computer Music Conference (ICMC), Beijing, China (1999)Google Scholar
  11. 11.
    de Laubier, S., Goudard, V.: Meta-Instrument 3: a look over 17 years of practice. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), Paris, France, pp. 288–291 (2006)Google Scholar
  12. 12.
    Egloff, D., Frissen, I., Wanderley, M.M.: Vibrotactile melodic interval discrimination on the index finger of the non-dominant hand. In: Proceedings of the Eurohaptics conference, London, UK (2016)Google Scholar
  13. 13.
    Enriquez, M., MacLean, K.E.: The role of choice in longitudinal recall of meaningful tactile signals. In: IEEE Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Reno, NV, USA, pp. 49–56 (2008)Google Scholar
  14. 14.
    Essl, G., O’Modhrain, S.: Scrubber: an interface for friction-induced sounds. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), Vancouver, Canada, pp. 70–75 (2005)Google Scholar
  15. 15.
    Frid, E., Giordano, M., Schumacher, M.M., Wanderley, M.M.: Physical and perceptual characterization of a tactile display for a live-electronics notification system. In: Proceedings of the Joint International Computer Music Conference and Sound and Music Computing Conference (ICMC–SMC), Athens, Greece (2014)Google Scholar
  16. 16.
    Geldard, F.A.: Some neglected possibilities of communication. Science 131(3413), 1583–1588 (1960)CrossRefGoogle Scholar
  17. 17.
    Giordano, M.: Vibrotactile feedback and stimulation in music performance. Ph.D. thesis, McGill University, Canada (2016)Google Scholar
  18. 18.
    Giordano, M., Hattwick, I., Franco, I., Egloff, D., Frid, E., Lamontagne, V., TeZ, C., Salter, C., Wanderley, M.M.: Design and implementation of a whole-body haptic suit for “Ilinx”, a multisensory art installation. In: Proceedings of the Sound and Music Computing conference (SMC), Maynooth, Ireland, pp. 169–175 (2015)Google Scholar
  19. 19.
    Giordano, M., Wanderley, M.M.: Perceptual and technological issues in the design of vibrotactile-augmented interfaces for music technology and media. In: Oakley, I., Brewster, S. (eds.) Haptic and Audio Interaction Design (HAID). Lecture Notes in Computer Science, vol. 7989, pp. 89–98. Springer, Berlin, Heidelberg (2013)CrossRefGoogle Scholar
  20. 20.
    Goebl, W., Bresin, R., Galembo, A.: Touch and temporal behavior of grand piano actions. J. Acoust. Soc. Am. 118(2), 1154–1165 (2005)CrossRefGoogle Scholar
  21. 21.
    Grosshauser, T., Hermann, T.: Augmented haptics—an interactive feedback system for musicians. In: Altinsoy, M.E., Jekosch, U., Brewster, S. (eds.) Haptic and Audio Interaction Design (HAID). Lecture Notes in Computer Science (LNCS), vol. 5763, pp. 100–108. Springer, Berlin, Heidelberg (2009)CrossRefGoogle Scholar
  22. 22.
    Gunther, E.: Skinscape: a tool for composition in the tactile modality. Master’s thesis, Massachusetts Institute of Technology, USA (2001)Google Scholar
  23. 23.
    Gunther, E., O’Modhrain, S.: Cutaneous Grooves: composing for the sense of touch. J. New Music Res. 31(1), 369–381 (2003)CrossRefGoogle Scholar
  24. 24.
    Hattwick, I., Franco, I., Wanderley, M.M.: The Vibropixels: A Scalable Wireless Tactile Display System, pp. 517–528. Springer Int. Publishing, Cham (2017)CrossRefGoogle Scholar
  25. 25.
    Ignoto, P., Hattwick, I., Wanderley, M.M.: Development of a vibrotactile metronome to assist in conducting contemporary classical music. In: Proceedings of the International Conference on Human Factors in Robots and Unmanned Systems (AHFE), Los Angeles, CA, USA, pp. 248–258 (2018)Google Scholar
  26. 26.
    Karam, M., Russo, F.A., Branje, C., Price, E., Fels, D.I.: Towards a model human cochlea: sensory substitution for crossmodal audio-tactile displays. In: Proceedings of the Graphics Interface Conference, pp. 267–274. Canadian Information Processing Society (2008)Google Scholar
  27. 27.
    Karam, M., Russo, F.A., Fels, D.I.: Designing the model human cochlea: an ambient crossmodal audio-tactile display. IEEE Trans. Haptics 2(3), 160–169 (2009)CrossRefGoogle Scholar
  28. 28.
    Keane, M., Dodd, G.: Subjective assessment of upright piano key vibrations. Acta Acust. united Ac. 97(4), 708–713 (2011)CrossRefGoogle Scholar
  29. 29.
    Keele, S.W.: Attention and Human Performance. Goodyear Pub. Co (1973)Google Scholar
  30. 30.
    Kyung, K.U., Lee, J.Y.: Ubi-Pen: a haptic interface with texture and vibrotactile display. IEEE Comput. Graph. Appl. 29(1), 56–64 (2009)CrossRefGoogle Scholar
  31. 31.
    MacLean, K.E.: Designing with haptic feedback. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 783–788 (2000)Google Scholar
  32. 32.
    MacLean, K.E.: Foundations of transparency in tactile information design. IEEE Trans. Haptics 1(2), 84–95 (2008)CrossRefGoogle Scholar
  33. 33.
    MacLean, K.E., Enriquez, M.: Perceptual design of haptic icons. In: Proceedings of the Eurohaptics Conference, Dublin, Ireland, pp. 351–363 (2003)Google Scholar
  34. 34.
    Marshall, M.T.: Physical interface design for digital musical instruments. Ph.D. thesis, McGill University, Canada (2008)Google Scholar
  35. 35.
    Marshall, M.T., Wanderley, M.M.: Examining the effects of embedded vibrotactile feedback on the feel of a digital musical instrument. In: Proceedings of the Conference on New Interfaces for Musical Expressian (NIME), Oslo, Norway, pp. 399–404 (2011)Google Scholar
  36. 36.
    Merchel, S., Altinsoy, M.E., Stamm, M.: Tactile music instrument recognition for audio mixers. In: Proceedings of the Audio Engineering Society Convention (AES), London, UK (2010)Google Scholar
  37. 37.
    Michailidis, T., Berweck, S.: Tactile feedback tool: approaching the foot pedal problem in live electronic music. In: Proceedings of the International Computer Music Conference (ICMC), Huddersfield, UK (2011)Google Scholar
  38. 38.
    Michailidis, T., Bullock, J.: Improving performers’ musicality through live interaction with haptic feedback: a case study. In: Proceedings of the Sound and Music Computing Conference (SMC), Padua, Italy (2011)Google Scholar
  39. 39.
    Miranda, E.R., Wanderley, M.M.: New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. The Computer Music and Digital Audio Series, A-R Edn. (2006)Google Scholar
  40. 40.
    Nichols, C.: The vBow: development of a virtual violin bow haptic human-computer interface. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), pp. 1–4 (2002)Google Scholar
  41. 41.
    O’Modhrain, S., Essl, G.: PebbleBox and CrumbleBag: tactile interfaces for granular synthesis. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), pp. 74–79 (2004)Google Scholar
  42. 42.
    O’Modhrain, S., Gillespie, R.B.: The moose: a haptic user interface for blind persons. In: Proceedings of the International World Wide Web Conference WWW6, Santa Clara, CA, USA (1997)Google Scholar
  43. 43.
    Puckette, M., Settel, Z.: Nonobvious roles for electronics in performance enhancement. In: Proceedings of the International Computer Music Conference (ICMC), Tokyo, Japan, pp. 134–134 (1993)Google Scholar
  44. 44.
    Qian, H., Kuber, R., Sears, A.: Tactile notifications for ambulatory users. In: Proceedings of the CHI’13 Extended Abstracts on Human Factors in Computing Systems, pp. 1569–1574. ACM, Paris, France (2013)Google Scholar
  45. 45.
    Schumacher, M., Giordano, M., Wanderley, M.M., Ferguson, S.: Vibrotactile notification for live electronics performance: a prototype system. In: Proceedings of the Internationa Symposium on Computer Music Multidisciplinary Research (CMMR), Marseille, France (2013)Google Scholar
  46. 46.
    Sinclair, S., Wanderley, M.M., Hayward, V.: Velocity estimation algorithms for audio-haptic simulations involving stick-slip. IEEE Trans. Haptics 7(4), 533–544 (2014)CrossRefGoogle Scholar
  47. 47.
    Sinyor, E., Wanderley, M.M.: GyroTyre : A dynamic hand-held computer-music controller based on a spinning wheel. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), Vancouver, Canada, pp. 42–45 (2005)Google Scholar
  48. 48.
    Tsukada, K., Yasumura, M.: ActiveBelt : Belt-type wearable tactile display for directional navigation. In: Ubiquitous Computing, Lecture Notes in Computer Science (LNCS), pp. 384–399. Springer (2004)CrossRefGoogle Scholar
  49. 49.
    van der Linden, J., Schoonderwaldt, E., Bird, J., Johnson, R.: MusicJacket—Combining motion capture and vibrotactile feedback to teach violin bowing. IEEE Trans. Instrum. Meas. 60(1), 104–113 (2011)CrossRefGoogle Scholar
  50. 50.
    van Erp, J.B.F., van Veen, H.A.H.C., Jansen, C., Dobbins, T.: Waypoint navigation with a vibrotactile waist belt. ACM Trans. Appl. Percept. 2(2), 106–117 (2005)CrossRefGoogle Scholar
  51. 51.
    Verrillo, R.T.: Vibration sensation in humans. Music Percept. 9(3), 281–302 (1992)CrossRefGoogle Scholar
  52. 52.
    Visell, Y.: Tactile sensory substitution: models for enaction in HCI. Interact. Comput. 21(1–2), 38–53 (2009)CrossRefGoogle Scholar
  53. 53.
    Von Békésy, G.: Similarities between hearing and skin sensations. Psychol. Rev. 66(1), 1–22 (1959)CrossRefGoogle Scholar
  54. 54.
    Wanderley, M.M., Depalle, P.: Gestural control of sound synthesis. Proc. IEEE 92(4), 632–644 (2004)CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  • Marcello Giordano
    • 1
    Email author
  • John Sullivan
    • 1
  • Marcelo M. Wanderley
    • 1
  1. 1.IDMIL—Input Devices and Music Interaction LaboratoryCIRMMT—Centre for Interdisciplinary Research in Music Media and Technology, McGill UniversityMontréalCanada

Personalised recommendations