1 Introduction

Practicing a musical instrument is a rich multisensory experience. As explained in Chap. 2, the instrument and player form a complex system of sensory-motor interactions where the sensory feedback provided by the instrument as a response to a playing action (bowing, plucking, striking, blowing, pumping, rubbing, fingering) is shaped not only by listening to the sound produced by that action, but also by feeling the cutaneous vibrations (vibrotactile sensation) and reactive forces (proprioceptive sensation) resulting from the same action. In assessing the heard sound in terms of technical execution and expressive intention—pitch, timing, articulation, dynamics, timbre—the musician integrates additional haptic cues before the next sound is made in order to adjust their playing technique. In this sense, the perception and evaluation of the quality of a musical instrument, as seen from the perspective of the performer, are a rich multisensory experience as well.

The proprioceptive component of the haptic feedback at a musical instrument is connected to the behavior of the instrument’s (re)action. An instrument with a precise and responsive action allows a skilled musician to produce a wide variety of timbre nuances through fine-grained control of synchrony, dynamics, attack speed, articulation, and balance in polyphonic texture. Vibrotactile feedback, on the other hand, consists essentially of the same oscillations that the instrument body radiates as sound [42, 49, 69,70,71] and is perceived simultaneously with the auditory signal, but differently [4, 6, 18, 25, 31, 41, 45, 62, 65]. In contrast to hearing, where maximal sensitivity is in the range of 3000–4000  Hz, vibrotaction is most sensitive in the vicinity of 250  Hz (see Sect. 4.2), which is within the range of most orchestral instruments and already at about 1000 Hz the sensation of vibrations is lost, whereas the range of most instruments extends well beyond this frequency. Tactile waveforms of varying type and complexity can be discriminated [1, 8, 51, 59, 72] and can activate areas of the auditory cortex in the absence of sound input [14]. Auditory and tactile frequency is likely calculated in an integrated fashion during preattentive sensory-perceptual processing—much earlier in the information processing chain than had been supposed [13]. An overview of further comparisons between the auditory and tactile modalities is given in Sect. 12.2. But is the vibrotactile sensation at a musical instrument perceptually relevant to its judged quality?

In the first part of this chapter, we will review recent research on the perceptual evaluation of violin quality from the perspective of the musician. Haptic feedback is particularly relevant in playing an instrument such as the violin where physical contact with the performer is highly intimate compared to other instruments due to the violin’s sound making mechanism. The fingers, chin, and shoulder of the violinist are in immediate contact with the vibrating parts of the instrument, implying a rich source of haptic feedback, an understanding of which should help to reveal particular aspects of quality perception. We will initially discuss psycholinguistic evidence of how violin quality is conceptualized in the mind of the violinist during playing-based preference tasks and then describe a series of studies on the perception and quality evaluation effects of vibrotactile feedback at the left hand of the violinist in normal playing scenarios.

Alongside the violin, we have chosen the piano as a second example case. Here, the contact between the performer and the instrument is much less intimate compared to the violin. Traditional piano playing involves touching only the keys (modern piano repertoire may sometimes require hitting or plucking the strings) and pedals (mediated by shoes). The nature and origin of piano touch have long been a source of fundamental disagreement in music performance and perception research: Are the timbre and loudness of a single note determined solely by the velocity of the hammer, or can the pianist further control them through the type of touch? In the second part of this chapter, we will then review recent literature on haptic feedback when playing the piano, examining the relationship between touch and tone quality, and more generally the importance of vibrotactile feedback to the perceptual evaluation of piano quality by the performer.

2 Violin

The violin as we know it today was developed in the early sixteenth century around Cremona in Italy and can be seen as the result of applying the tuning of the medieval rebec (fifths) to the body of the lira da braccio [16]. The transition from baroque to classical music led to a few further modifications in the second half of the eighteenth century, such as a longer, narrower fingerboard, and neck. Since then, the basic violin lutherie has remained largely unchanged, combining visual charm with ergonomics and a precise acoustical function.

Sound is produced by bowing (or plucking) one or more strings at a location between the bridge and the edge of the fingerboard. The played string produces oscillations that are not efficiently radiated by the string itself due to its much smaller diameter than the acoustic wavelength of most audible frequencies [23]. Instead, the forces exerted from the vibrating string on the bridge cause the violin body to vibrate and thus radiate sound. The varying patterns in which different harmonics are transformed by the vibrating modes (resonances) of the body thus “color” the radiated sound. Figure 5.1 depicts a typical violin frequency response function (defined as the input admittance measured at the E-string notch on the bridge). Furthermore, violin body resonances exhibit a slow decay that brings a “ringing” quality to the sound [37]. At frequencies above about 1 kHz, the motions of the body create frequency-dependent directivity formations that add “flashing brilliance” to its sound [64].

Fig. 5.1
figure 1

Input admittance of a violin obtained by exciting the G-string corner of the bridge with a miniature force hammer and measuring the velocity at the E-string corner of the bridge with a laser Doppler vibrometer [52]. The magnitude and phase are shown in the top and bottom plots, respectively. Some of the so-called signature modes (i.e., strongly radiating and thus crucial to violin sound) can be observed in the open string region, below about 600  Hz: the Helmholtz-type cavity mode A0 at around 280 Hz and the first strongly radiating corpus bending mode B1\(^+\) just above 500  Hz. Also, important is the hill-like collection of peaks known as the “BH peak” (bridge and/or body hill) in the vicinity of 2–2.5 kHz, which allows a solo violin to be heard over an ensemble of instruments

Full size image

2.1 Touch and the Conceptualization of Violin Quality by Musicians

Attempts to quantify the characteristics of “good” and “bad” violins from vibrational measurements such as the input admittance (Fig. 5.1) and/or listening tests have largely been inconclusive (see [52] for a review). On the one hand, this may be due in part to overly broad characterizations of “good” and “bad.” On the other hand, both approaches end up considering the instrument isolated from the musician and no haptic information is provided. Woodhouse was among the first to consider that what distinguishes one violin from another lies not only in its perceived sound quality but also in what he termed its playability, as in how the violinist “feels” the instrument and how easy it is to produce a good sound [68]. To this end, recent research on violin acoustics and quality has focused attention on the perceptual and cognitive processes involved when violinists assess violins under normal playing scenarios.

Fritz and colleagues carried out a series of listening tests using virtual violins, whereby synthesized bridge-force signals were convolved with a digital filter mimicking the input admittance of the violin [29]. The measured admittance of a “good-quality” modern violin was first decomposed into its modal components, the parameters of which were then used to re-synthesize it, allowing for controlled variations of vibrato and body damping. Results showed that when listening to single notes, violinists found it difficult to assess the “liveliness” of the sound, and often, the word itself was not used in a consistent way across individuals. But when asked to play on an electric violin, whereby the actual bridge-force signal was passed through modified re-synthesized admittances in real time, musicians were able to rate liveliness consistently within and between individuals. This seems to suggest that liveliness is processed differently in passive listening versus active playing contexts, where haptic cues from proprioceptive and vibrotactile feedback are present.

In another study, preference judgments made by three violin players during a listening and a playing test were compared in conjunction with psycholinguistic analyses of free-format verbal descriptions of musician experience provided by the three violinists [28]. The authors used a method from cognitive linguistics that relies on theoretical assumptions about cognitive-semantic categories and how they relate to natural language [20]. Categories can be thought of as collective representations and knowledge, to which individual assessments are conveyed by means of a shared discourse. From what is being said (content analysis) and how it is being said (linguistic analysis), relevant inferences about how people process and conceptualize sensory experiences can be derived (semantic level) and further correlated with physical parameters (perceptual level). This approach has been applied to other instruments such as the piano [11] and the guitar [50], providing novel insights into how musicians perceive instrumental sound as well as playing characteristics. Fritz and colleagues found that the overall evaluation of a violin, as reflected in the verbal responses of the musicians, varied between listening and playing conditions, and the latter invoking linguistic expressions influenced not only from the produced sound but also by the physical interaction between the performer and the instrument.

Saitis and colleagues carried out two violin playing perceptual tests based on a carefully controlled protocol [56, 57]. Emphasis was given to the design of conditions that are musically meaningful to the performer (e.g., playing versus listening, comparing different instruments like in a violin workshop, using own bow, allowing time to familiarize with the different violins, developing own strategy). In the first experiment, skilled violinists ranked a set of different violins from least to most preferred. In the second experiment, another group of players rated a different set of violins according to specific attributes as well as preference. In both experiments, musicians were asked to verbally describe their choices through open-ended questions. Analyses of intra-individual consistency and inter-player agreement in the (nonverbal) preference and attribute judgments showed that while violinists generally agreed on what particular attributes they look for in an instrument, the perceptual evaluation of the same attributes varied dramatically across individuals, thus resulting in large inter-player differences in the preference for violins. A third experiment [58] and studies by Fritz et al. [26, 27] and Wollman et al. [66, 67] reached similar conclusions.

To better understand the perceptual and cognitive processes involved when violinists evaluate violins, Saitis and colleagues further analyzed the verbal expressions collected in their two violin playing tests [53,54,55], expanding on an earlier work of Fritz et al. [28]. Based on psycholinguistic inferences, it was argued that violin players of varying style and expertise share a common framework for conceptualizing violin quality on the basis of semantic features and psychological effects that integrate perceptual attributes (i.e., perceptual correlates of physical characteristics) of not only the sound produced but also the vibrotactile and proprioceptive sensations experienced when playing the instrument (Fig. 5.2). The bowed string and vibrating body system contribute to the perception of sound quality through (a) the amount of felt vibrations in the left hand, shoulder, and chin (conceptualized as resonance); (b) through assessing the offset (speed) and amount (ease) of reactive force (conceptualized as response) from the body in the right hand (through the bow) with respect to the quality and intensity of the heard as well as felt vibrations; and (c) through comparing these between different notes across the instrument’s register (conceptualized as balance across strings).

These psycholinguistic investigations provide empirical evidence that vibrations from the violin body and the bowed string (via the bow) are used by violinists as extra-auditory cues that not only help better control the played sound [4], but also contribute to a crossmodal audio-tactile assessment of its attributes. The perception of violin sound quality is thus elaborated both from sensations linked to auditory information and from haptic factors associated with proprioceptive and vibrotactile cues. The cognitive model shown in Fig. 5.2 raises interesting questions concerning the characterization of haptic feedback in violin playing quality tests—what to measure and how? Can standard vibrational measurements, such as a violin’s bridge admittance (Fig. 5.1), capture everything significant about the reactive force and vibration levels felt by the player? If yes, in what ways can this information be extracted?

Fig. 5.2
figure 2

From body vibrations to semantic categories: a cognitive model describing how the perception of violin quality is elaborated on the basis of both auditory and haptic cues [55]

Full size image

2.2 Vibrotactile Feedback at the Left Hand

Acoustics and psychophysics literature on the “feel” of a violin has been limited compared to the ample amount of research on the instrument’s sound. Marshall suggested that violin neck vibrations felt through the left hand form the basis for the perception of how a violin feels [43, 44]. He argued that the more often the left hand detects motions at antinodal parts of the neck (which are typically damped when the musician holds the violin but can be sensed directly on the skin), the more “alive” the violin will be felt. Askenfelt and Jansson showed that vibrations perpendicular to the side of the neck, measured on four violins of varying quality during playing a single note (lowest G, 196 Hz), were above or very close to vibration sensation thresholds measured at the fingertip under passive touch conditions by Verrillo [61]. However, no evidence was found that higher neck vibration intensity would result in judging a violin as being of better quality [4]. One limitation of that study was that vibration amplitude was measured for five frequencies only, corresponding to the first five harmonics of the played note and thus lying below the 1 kHz upper limit of the human skin sensitivity range. Another potential issue—discussed in Sect. 4.3.4 for the piano—is that Verrillo’s thresholds may not fully reflect actual vibration detection offsets when the left hand holds the neck of the violin (e.g., differences in location and size of contact area, pressure exerted from the hand on the neck).

Fig. 5.3
figure 3

Reproduced from [65] with permission from S. Hirzel Verlag

Horizontal vibration levels at the side of the necks of violins (first position) perceived as either a “vibrating” or b “non-vibrating” (solid lines) and vibration sensation threshold at the left hand of violinists (dashed line).

Full size image

Wollman and colleagues were the first to systematically address the role of haptic cues from neck vibrations on violin quality perception. Expanding on the work of Askenfelt and Jansson [4], vibration levels were measured at the violin neck in first positionFootnote 1 across a set of ten instruments, which were characterized by a professional violinist according to how “vibrating” they were felt to be [65]. Neck vibration frequency response curves of “vibrating” and “non-vibrating” violins, obtained across the whole range of the instrument through laser vibrometry, were then compared to absolute vibrotactile thresholds measured on fourteen violinists holding in first position a real isolated violin neck vibrating at six frequencies between 196 and 800  Hz (the first four were chosen to correspond to the open strings). This setup helped obtain violin playing-specific thresholds (i.e., measured under active touch conditions, similar to what was done in Sect. 4.3 for the piano) that are more appropriate to compare with vibration levels than those measured by Verrillo [61] and used by Askenfelt and Jansson. It was observed that while neck vibrations of “vibrating” violins were well above the detection threshold by an average of 15 dB in the range 200–800  Hz, those of “non-vibrating” violins exhibited a steep attenuation of about 40 dB around 600 Hz and stayed below or close to the threshold above that (Fig. 5.3).

Fig. 5.4
figure 4

Reproduced with the permission of the Acoustical Society of America

Experimental setup for transmitting vibrations from the neck of a played violin to an isolated neck [66].

Full size image
Fig. 5.5
figure 5

Reproduced from [66] with the permission of the Acoustical Society of America

Mean comparison ratings of three violins (V1–3) across several quality criteria between two different levels of vibration (full versus reduced by half). A positive score indicates a higher rating when full vibrations are present than when reduced by half.

Full size image

In another study [66], fifteen professional musicians listened to three violins while seating on a chair and holding a real isolated violin neck on which they fingered the performed score. The instruments were being played live by another violinist (non-participant) in the same room, placed behind a curtain in front of the participants. Along with the live sound, vibrations of the played violins were picked up at the scroll using a small accelerometer and then transmitted through a shaker system to the isolated neck (Fig. 5.4). They were presented either at the same level as in the played violin, reduced by half, or fully attenuated. This condition was described by the authors as active listening. Participants were asked to rate the violins on richness of sound, loudness, responsiveness, and pleasure of playing. It was observed that violinists judged all three violins as having a less loud but also a less rich sound whenever the level of vibrations felt on the isolated neck was reduced by half (Fig. 5.5). These results complemented the findings of Yau and colleagues, who have shown that in a non-musical context, the simultaneous presentation of tactile distractors can cause an increase in perceived tone loudness [71].

Fig. 5.6
figure 6

Reproduced from [67]; published under the Creative Commons Attribution (CC BY) license

Mean preference ratings of five violins under three different playing conditions (COND): normal (N), masked auditory feedback (noA), masked tactile feedback (noT). Vertical bars represent the standard errors of the mean.

Full size image

In a third experiment [67], twenty violinists evaluated five violins under three sensory masking conditions: playing without hearing the produced sound, playing without feeling the produced vibrations, and playing normally (i.e., neither modality was masked). Auditory feedback was masked by means of earmuffs and in-ear monitors playing white noise with a bandwidth of 20–20000  Hz, while passive anti-vibration material was added to the chin rest to minimize bone conduction. Vibrations were primarily masked on the left hand using vibrating rings worn on the thumb, index, and ring fingers, while vibrations through the chin and shoulder rests were attenuated as in the auditory masking scenario. In each condition, musicians first rated each violin on a number of criteria related to perceived sound and playing characteristics and then commented on how relevant those criteria were each time. These data provided further evidence that the perceptual evaluation of violin attributes such as liveliness, power, evenness across the strings, or dynamic range relies not only on sonic information but also on vibrotactile cues. Concerning overall preferences, it was observed that removing auditory feedback was not more disruptive than attenuating felt vibrations, although its effect tended to depend on the instrument (Fig. 5.6).

These studies indicate that the violin neck vibrations felt by violinist through the left hand can serve as an important cue to the concept of “feel” in violin quality evaluation, as well as augment the perception of qualities attributed to the sound (in that case “loud” and “rich”). They also introduce novel methods for characterizing vibrotactile feedback at the left hand. Another source of haptic cues that potentially relate to perceived “feel” and sound quality is the vibration of the chin rest. Askenfelt and Jansson argued that the jaw is less sensitive than the left hand, but it may still be possible for the violinist to sense these vibrations because of the larger contact area of the jaw with the chin rest [4]. Similarly to the violin neck, it would be interesting to investigate whether vibrotactile feedback at the chin contributes to the perception of a violin’s “feel” and/or sound.

Fig. 5.7
figure 7

Reprinted from [3] with the permission of the Acoustical Society of America

Illustration of the function of the piano action at successive stages during a keystroke. a Rest position: The hammer rests via the hammer roller on the repetition lever, a part of the lever body. The lever body stands on the key, supported by the capstan screw. The weight of the hammer and lever body holds the playing end of the key in its upper position. The damper is resting on the string. b Acceleration: When the pianist’s finger depresses the key, the lever body is rotated upward. The jack, mounted on the lever body, pushes on the roller and accelerates the hammer. The damper is lifted off the string by the inner end of the key. c Let-off: The tail end of the jack is stopped by the escapement dolly, and the top of the jack is rotated away from the hammer roller. The free hammer continues toward the string. The repetition lever is stopped in waiting position by the drop screw. d Check: The rebounding hammer falls with the hammer roller on the repetition lever in front of the tripped jack. The hammer is captured at the hammer head by the check at the inner end of the key.

Full size image

3 Piano

The modern piano, descending from the harpsichord and introduced by Bartolomeo Cristofori in 1709, evolved into two distinct types, the grand piano and the upright piano. The latter was developed in the middle of the nineteenth century, and its action differs somewhat from that of the first due to design constraints, although they share the same sound production principle [23]: A piano string is set in vibration when the respective key is depressed, a damper raised, and a felt hammer hits the string (Fig. 5.7). String vibrations are transmitted through the bridge to the soundboard, from which the sound radiates into the air. Modal structure of the soundboard and material properties further contribute to the acoustics of the piano. The sound is characterized by different decay rates between partials [21], a two-part pattern of time decay (or double decay) due to double and triple unison strings [63], and inharmonicity in terms of stretching of the partials due to string stiffness [22].

3.1 Piano Touch and Tone Quality

There is a long-standing discrepancy between the acoustical basis of how the timbre of a single piano tone is created and the practical experience of piano performers [3, 5]. When considering only the mechanics of the hammer-string interaction, piano timbre would be an instrument-specific result of loudness, which in turn depends on the velocity at which the hammer hits the string, controlled only through key velocity produced by the finger pressing force of the player. The way of touching the key would therefore have no influence on the resulting timbre. Skilled pianists, on the other hand, aim to control timbre and loudness independently through touch and gestural means involving movements of the entire upper body. A review on the historical development of various schools on piano technique as well as recent performance analysis and biomechanical studies on piano touch is presented by MacRitchie [40].

There is some evidence in favor of the touch effect, although it seems to be weaker than many pianists believe and mostly caused by other aspects of the sound than the tonal component. Goebl and colleagues measured the ability of pianists to perceive differences in piano sound independently of intensity [35]. Half of the participants were able to correctly distinguish between struck and pressed touch in the presence of finger-key noises occurring 20–200 ms before the sound. When the noises were cut from the sound signals, performance dropped to chance level. Pianists were also able to distinguish piano sounds of equal hammer velocity with either present or absent key-keybed noises with an average of 82% accuracy [34]. Askenfelt observed that structure-born transients, dependent on the type of touch and present 20–30 ms before the first transversal wave on the string arrives at the bridge, may potentially be connected with the pianist’s touch [2]. More recently, numerical simulations of the hammer head-shank interaction showed a difference in spectral profile between legato and staccato sounds in the range of 500–1000  Hz [17]; however, an effect on perceived timbre was not shown experimentally. Suzuki reported a slight spectral brightening for G5, in the order of 1.5 dB at the tenth partial, as a result of “hard” or “soft” touch depending on the degree of stiffness of shoulder, elbow, wrist, and finger [60]. When listening only, about half of the participants could distinguish an effect of similar degree after training.

To discover how pianists achieve fine-grained control of their instrument’s sound, the way they describe and recognize timbre nuances in piano performance has gained interest. Bernays and Traube quantified a semantic space of five descriptors (dry, bright, round, velvety, and dark) [10] based on an analysis of free verbalizations provided by pianists [7] and conducted a series of studies where pianists performed pieces highlighting each of the five semantic dimensions of piano timbre. Despite differences between musicians relating to individual playing styles, common timbre nuance strategies were revealed across different performances [11, 12]. The latter were saliently grouped by the intended timbre on a bidimensional space by means of principal components analysis. The first component was found to be associated with dynamics, attack, and soft pedal features, while the second dimension was related to sustain pedal. Further playing style factors included key depression depth, legato versus staccato articulation, and balance between hands.

Given the pianist’s common ways of nuance control, the question arises whether listeners can differentiate and identify the resulting timbres in piano performance. To this end, Bernays reported a pilot study where listeners both described freely and identified in a forced choice task the timbre of piano performance excerpts, each intended to reflect one of the following timbre nuances: bright, dark, distant, full-bodied, harsh, matte, round, and shimmering [9]. Participants identified the timbre categories above chance level except for round and matte. Some categories, like bright and shimmery, were frequently mixed up, probably due to their semantic proximity.

These studies have revealed that pianists can control timbre independently of dynamics: The way of touching the keys produces differences in contact noises (finger-key, key-key bottom, and release sounds) as well as slight spectral effects. While these may be inaudible to the average listener, they have a stronger and more important effect on the skilled pianist due to sensory integration of the matching touch and sound information [15]. Especially in polyphonic touch, these subtle vibrotactile cues may enable the player to produce and control a wide range of timbre nuances.

3.2 Haptic Cues and Instrument Quality

Some early experiments on multimodal perception of piano quality were conducted by Galembo and Askenfelt [30], in which pianists evaluated four concert grand pianos under varying sensory feedback conditions. When freely playing the instruments, professional pianists ranked them as expected according to the manufacturers’ reputation. However, musicians failed to identify the pianos in a listening-only condition, nor was the resulting quality ranking equal to the playing-based evaluation. In a subsequent free playing task, where visual feedback was blocked by means of blindfolding, the musicians and auditory feedback was blocked through masking noise, the pianists were actually able to identify the instruments without difficulty. These experiments offer some evidence that pianos can be identified by their haptic response perhaps even better than by their sound. As an underlying mechanism, one should expect that different piano actions react differently to different dynamics and types of touch and that these differences are perceivable and possibly of more importance than auditory cues to the player.

Askenfelt and Jansson had previously made timing measurements of the various parts of the piano action and observed differences mainly as a function of dynamics and regulation of the action (mechanical adjustments to compensate for the effects of wear) [3]. Goebl et al. [36] studied in detail the temporal behavior of three grand piano actions. Touch-related differences were found through measurements of finger-key, hammer-string, and key-keybed contact times and maximum hammer velocities throughout the entire dynamic range for several keys. A different key velocity trajectory in struck and pressed sounds was also observed. Struck sounds showed two acceleration phases of key velocity, while the pressed sounds developed more linearly. These differences between struck and pressed touch were observed in all three pianos that were measured. However, it remains unknown how the behavior of the piano action may affect the player experience. The authors of the study hypothesize that since the pianist needs to (unconsciously) estimate the path from touch to tone onset and intensity for various dynamics and types of touch, a high-quality instrument is one that has a precise and consistent action. In their own informal evaluation as pianists, the most highly appreciated instrument turned out to have the lowest compressibility of the parts of action, short free-travel times of the hammer, and late maxima in the hammer velocity trajectory.

3.2.1 Vibrations in the Acoustic Piano

Keane analyzed keyboard vibrations at four upright and four grand pianos by removing harmonic peaks from the spectrum of the vibration signal and thus splitting it into tonal and broadband parts [38]. Similar tonal components were observed across the two piano types, but upright pianos showed a stronger broadband component, which could explain the generally lower perceived quality of upright versus grand pianos. In fact, a later study showed that pianists preferred the tone quality and loudness profile of an upright piano with attenuated broadband vibrations [39].

Fontana and colleagues investigated the effect of key vibrations on acoustic piano quality using both a grand and an upright Yamaha Disklavier, which can operate in both an acoustic and silent mode [25]. While playing, pianists received auditory feedback through a piano software synthesizer and tactile feedback through the Disklavier keyboard. The technical setup is described in more detail in Sect. 4.3.1. The experimental task involved comparing a non-vibrating to a vibrating piano setup during free playing according to several quality attributes. In the non-vibrating setup (A), the Disklavier was operating in silent mode, which prevents the hammers from hitting the strings and thus from producing vibrations. In the vibrating setup (B), the Disklavier was operating in acoustic mode, allowing the natural vibration of the strings to be transmitted to the soundboard as well as to the keys. However, the acoustically produced sound was blocked by insulating earmuffs placed on top of the earphones playing back the synthetic piano sound. Pianists rated the following attributes on a continuous scale ranging from −3 (“A much better than B”) to +3 (“B much better than A”): dynamic range, loudness, richness, naturalness, and preference. All attributes except preference were rated separately in the low (keys below D3), mid (keys between D3 and A5), and high (keys above A5) range.

Fig. 5.8
figure 8

Results of the piano quality experiment described in [25]: Means with errorbars ± SE as given by [46]. Positive values signify preference of the vibrating mode. The labels on the x-axis show short names for the evaluated quality features (dyn = dynamic range, lou = loudness, rich = richness, nat = naturalness, and pref = preference) and the considered keyboard ranges (l = low, m = medium, h = high)

Full size image

For both the grand and the upright piano type, the vibrating setup was preferred to the non-vibrating condition (Fig. 5.8). The mean preference scores were 1.05 (\(n=15\), SD \(=1.48\)) for upright piano and 0.77 (\(n=10\), SD \(=1.71\)) for grand piano. The distributions of the preference ratings did not differ significantly between pianos. Interestingly, while the participants generally preferred when vibrations were present, in the subsequent debriefing only one of them could pinpoint vibration as the difference between the setups. There was considerable positive correlation between attribute scales and frequency ranges. Ratings correlated highly between the low and mid ranges (mean Pearson \(\rho =0.58\)) and between the mid and high regions (\(\rho =0.43\)). At a later stage, a vibration detection sensitivity experiment conducted using the same setup (see Sect. 4.3) showed that piano key vibrations are perceived roughly up to note A4 (440  Hz). As such, the high range was entirely beyond the sensitivity range. That said, the detection experiment was performed under controlled timing and single notes or three-note clusters in the high range, while a free playing task constitutes a more ecological setting (usually involving multifinger interaction). This may explain the slight effect of vibration on higher frequencies in the latter. For further analysis, new dependent variables were formed by taking the average over the low- and mid-frequency ranges. Partial correlation analysis and principal components analysis suggested that naturalness and richness of tone were the attributes most associated with increased preference.

Fig. 5.9
figure 9

Positive and negative ratings in the piano quality experiment described in [25]

Full size image

Inter-individual consistency was low in both piano groups, suggesting high disagreement between individuals. Specifically, five participants preferred the non-vibrating setup. When the negative preference rating was used as a criterion for a posteriori segmentation [48], the attitudes of the two groups segregated clearly. While the negative and positive groups gave rather similar ratings for dynamic range and loudness, their mean ratings for richness, naturalness, and preference were clearly different (Fig. 5.9). The mean preference ratings were 1.58 (\(n=20\), SD \(=0.79\)) and \(-1.61\) (\(n=5\), SD \(=1.10\)) for the positive and negative groups, respectively. Thus, while 80% of the participants associated dynamic range and loudness with naturalness, richness, and preference, the remaining 20% had the opposite opinion.

3.2.2 Digital Piano Augmented with Vibrations

A recent study on the effect of the nature of vibration feedback on perceived piano sound quality suggested that pianists may well be sensitive to the match between the auditory and the vibrotactile feedback [24]. The experimental setup (described in detail in Sect. 13.3.2) involved a digital keyboard enhanced both by realistic and synthetic key vibrations. Realistic vibrations were recorded from a Yamaha Disklavier grand piano. Synthetic vibration signals were generated using bandpass-filtered white noise, centered at the pitch and matching the amplitude envelope and energy of the recorded vibrations. They were interpolated according to key velocity and reproduced by transducers attached to the bottom of a digital piano. The reference setup consisted of auditory feedback only (A). The three test setups consisted of auditory feedback plus (B) recorded real vibrations, (C) recorded real vibrations with 9 dB boost, and (D) synthetic vibrations. Each of the test setups was compared to the reference setup in a free playing task, similar to what described above for the acoustic piano. Ratings were given on dynamic control, richness, engagement, naturalness, and overall preference.

Fig. 5.10
figure 10

Results of the digital piano quality experiment described in [24]. Boxplot presenting median and quartiles for each attribute scale and vibration condition. Positive values indicate preference for the vibrating setup

Full size image

On average, participants preferred the vibrating setup in all categories except for naturalness in condition D (Fig. 5.10). The strongest preferences were for dynamic control and engagement. Generally, condition C was the most preferred of the vibration conditions: It scored highest on four of the five scales, although B was considered the most natural. Interestingly, B scored lowest in all other scales. Similar to the Disklavier experiment discussed in the previous section, participants could be classified a posteriori into two groups, where median preference ratings for setup C were +2.0 and −1.5 for each group, respectively. In the larger group of positive preference (\(n=8\)), nearly all attributes were rated positively versus only one in the smaller, negative group (\(n=3\)). Notably, although auditory feedback remained unchanged, participants associated higher preference of the vibrating setup to richness of tone, which, during preparation for the experiment, was explained to them as a sound-related attribute. This supports the hypothesis that from the perspective of the musician, the perception of instrument quality emerges though the integration of both auditory and haptic information.

4 Conclusions

The perceptual evaluation of musical instrument quality has traditionally been considered a unisensory experience in the scientific and industrial world alike, based exclusively on how the produced tone sounds in terms of pitch, dynamics, articulation, and timbre. To a certain extent, this is naturally expected. After all, the objective of playing a musical instrument is to make (musical) sounds. But while this holds true for the non-musician listener, it only tells part of the story from the perspective of the musician, where aural impression is accompanied by haptic feedback due to one or more bodily parts of the player physically touching vibrating components of the instrument. Well-established theories of sensory-motor multimodal interaction and auditory-tactile multisensory integration in the analytical and empirical study of music performance assert that haptic cues carry important information concerning the control of the (re)action of the instrument and thus its sound and that temporal frequency representations are perceptually linked across audition and touch.

The violin and the piano offer unique example cases to examine whether the haptic interaction between the musician and the instrument can have a perceptual effect on quality evaluation. Both instruments require a significant amount of sensory-motor synergy to produce refined and precise sonic events, providing rich haptic feedback to the performer. At the same time, unlike the piano setup, violinists experience vibrations at other bodily parts than the hands, which makes it difficult to measure performance parameters and control vibrotactile feedback in normal playing experimental scenarios. The physical differences in the violin versus piano touch and the experimental freedoms or constraints imposed by them can help better understand the role of vibrotaction on the playing experience as well as the expressive possibilities it can afford in varying performance contexts. Particularly in the case of the piano, the MIDI protocol and the availability of computer-controlled keyboard instruments such as the Yamaha Disklavier and Bösendorfer CEUS offer fertile opportunities to obtain detailed piano performance data under well controlled but musically meaningful experimental conditions, although with some limitations [33].

Our review has shown that the vibrotactile component of the haptic feedback during playing, both for the violin and the piano, provides an important part of the integrated sensory information that the musician experiences when interacting with the instrument. In particular, the most recent violin and piano studies provide evidence that vibrations felt at the fingertips (left hand only for the violinist) can lead to an increase in perceived sound loudness and richness, suggesting the potential for more research in this direction. Investigations of the type and role of musical haptic feedback have also been reported for other instruments (e.g., [19, 31, 32]) as well as singing [47]. A vast field of topics await investigation, starting from the methods and aspects of instrument quality evaluation per se [15]. In which aspects does haptic feedback have a significant role? Which performance parameters (for example, timing accuracy) can be used to assess the haptic dimension in instrument quality perception?