1 Introduction

For what we have seen in Chap. 3, the somatosensory system relies on input from receptors that operate within deformable human tissues. One solution for measuring their activity precisely is to keep those tissues free from any kinematic perturbation. Such experiments—in which subjects were typically stimulated with vibrations at selected areas of their skin while remaining still—have set the roots of the psychophysics of passive touch. However, as Gibson observed in 1962, “passive touch involves only the excitation of receptors in the skin and its underlying tissue,” while “active touch involves the concomitant excitation of receptors in the joints and tendons along with new and changing patterns in the skin” [24]. This observation suggests that the psychophysics of active touch may exhibit relevant differences from the passive case. Furthermore, a systematic investigation of active touch psychophysics presents additional practical difficulties in experimental settings due to interactivity, which seems to motivate the current lack of results in the field. Even if we assume a small and well-defined vibrating contact at the fingertip, any change in this contact—as typically found in finger actions such as sliding or pressing—gives rise to new normal and longitudinal forces acting on the skin and to different contact areas. Such side-effects are indeed known to alter the tactile percept [9, 10, 28, 34, 36, 54]. The surrounding skin regions, which contribute to tactile sensations, are also dynamically affected by such changes and by the patterns of vibrations propagating across them [49].

The perception of vibrations generated by musical instruments during playing does not make an exception to the above mechanisms. In fact, the respective experimental scenario is conceptually even more complicated and technically challenging. While in general tactile stimuli may be controlled reasonably well in active touch psychophysics experiments, when considering instrumental performance one has to take into account that vibrations are elicited by the subjects themselves while playing and that concurrent auditory feedback may affect tactile perception [30, 46, 50, 59].

As explained in Chap. 2, a tight closed loop is established between musicians and their instruments during performance. Experimentation on active touch in the context of musical performance hypothesizes that tactile feedback affects such interaction in a number of ways and eventually has a role in the production of musical sounds.

1.1 Open-Loop Experimentation

The study of haptic properties of musical instruments outside of the musician–instrument interaction (i.e., in open loop) conceptually simplifies the experimental design, while effectively preparing the ground for further studies in closed loop.

The violin, due to its intimate contact with the player, represents one of the most fascinating instruments for researchers in musical haptics. A rich literature has grown to explain the physical mechanisms at the base of its range of expressive features [60]. However, the mechanical coupling of the violin with the performer is strong, so that its vibratory response measured in free-suspension conditions cannot fully represent the vibrotactile cues generated by the instrument when in use [38].

The vibratory response of the piano is relatively easier to assess, as the instrument’s interface with the musician is limited to the keyboard and pedals. Furthermore, the mass of the piano is such that the mechanical coupling with the performer’s limbs cannot affect its vibrations significantly. However, pianos couple with the floor; hence, vibrations can reach the pianist’s body through it and the seat. Piano vibrations have been carefully studied by researchers in musical acoustics, who measured them mainly at the strings or soundboard [51]. In contrast, keyboard vibrations as conveyed to the player have been less researched. In the early 1990s, Askenfelt and Jansson performed extensive measurements on several stringed instruments, including the double bass, violin, guitar, and piano [4]. Overall, vibration amplitude was measured above the standard sensitivity thresholds for passive touch [54], suggesting a role for tactile feedback at least in conveying a feeling of a resonating and responding object. This conclusion, though, was mitigated for the piano keyboard, whose vibration amplitude was mostly found below such thresholds and hence supposedly perceptually negligible. More recently, Keane and Dodd reported significant differences between upright and grand piano keyboard vibrations, while hypothesizing a perceptual role of vibrotactile feedback during piano playing [32].

Other classes of instruments, such as aerophones, likely offer measurable vibrotactile cues to the performer, but to our knowledge a systematic assessment of the perceivable effects of such vibratory feedback has not been yet conducted.

Percussion instruments, on the other hand, respond with a strong kinesthetic feedback that is necessary for performers to rearm their limbs instantaneously, and for executing rebounds and rolls without strain. In this regard, Dahl suggested that the interaction of a drumstick or a hand with the percussion point happens so rapidly, that it does not seem possible for a performer to adjust a single hit simultaneously with the tactile feedback coming from it [11]. The percussive action, in other words, appears to be purely feed-forward as far as multiple hit sequences are not considered (see also Sect. 2.2 in this regard). Finally, electroacoustic and electronic instruments do not seem able to generate relevant vibrotactile feedback, unless a loudspeaker system is mounted directly aboard them.

1.2 Experiments with Musicians

Once an instrument has been identified as a source of relevant tactile cues, their potential impact on musical performance and produced musical sound may be tested with musicians. The inclusion of human participants, however, introduces several issues. To start with, as mentioned above, interactive contexts such as the musical one prevent the implementation of experiments with full control over contact areas and forces, or the generation of vibratory stimuli. Also, acoustical emissions from musical instruments engage musicians in a multisensory process where the tactile and auditory channels are entangled at different levels, ranging from the peripheral and central nervous system, to cross-modal perceptual and cognitive processes. Tactile and auditory cues start to interfere with each other in the middle ear. Vibrations in fact propagate from the skin to the cochlear system through bone and tendon conduction, via several pathways [12]. Especially if an instrument is played close to the ear (e.g., a violin) or enters into contact with large areas of the body (e.g., a cello or double bass), such vibrations can reach the cochlea with sufficient energy to produce auditory cues. Cochlear by-products of tactile feedback may be masked by overloading the hearing system with sufficiently loud sound that does not correlate with tactile feedback: Masking noise provided through headphones is often necessary in tactile perception tests [6, 58]. The use of bone-conduction headphones may improve experimental control, as bone-conducted cues could be jammed on their way to the cochlea by vibratory noise transferred to the skull [47]. Even when considering only airborne auditory feedback, earmuffs or earplugs may not provide sufficient cutoff, and uncorrelated masking noise may be needed. The question, then, is how to analyze answers from musicians who had to perform while listening to loud noise. The literature on audio-tactile sensory integration is particularly rich and can help explain possible perceptual synergies or cancellations occurring during this integration [46, 50, 57, 58].

Any tactile interaction experiment that involves musicians should take the aforementioned issues into account. In a groundbreaking study from 2003, Galembo and Askenfelt showed that grand pianos are mainly recognized—and possibly even rated—based on the tactile and kinesthetic feedback offered by their keyboards, more than based on the produced sound [20]. Similarly, in a later study on percussive musical gestures, Giordano et al. showed that haptic feedback has a bigger influence on performance than on auditory cues [25]. Focusing on tactile cues alone, Keane and Dodd reported significant preference of pianists for an upright instrument whose keybed had been modified to decrease vibrations intensity at the keyboard, thus making them comparable to those produced by a grand piano [31, 32]. In parallel, some authors of the present chapter augmented a digital piano with synthesized vibrotactile feedback, showing that it significantly modified the performer’s preference [16, 18]. In the same period, one of the world’s top manufacturers equipped its flagship digital pianos with vibration transducers making the instruments’ body vibrate while playing [27], thus testifying concrete interest from the industry at least for the aesthetic value of tactile cues.

More recently, Wollman et al. showed that salient perceptual features of violin playing are influenced by vibrations at the violin’s neck [59], and Altinsoy et al. found similar results using reproduced vibratory cues [3]. Saitis et al. discussed the influence of vibrations on quality perception and evaluation as manifested in the way that musicians conceptualize violin quality [48]. Further details on the influence of haptic cues on the perceived quality of instruments are given in Chap. 5.

1.3 Premises to the Present Experiments

Compared to other interfaces of stringed instruments, the piano keyboard is easier to control experimentally, as the performer is only supposed to hit and then release one or more keys with one or more fingers. Other body contacts can be prevented by excluding the use of the pedals. Also, non-airborne auditory feedback—a by-product of the tactile response—can be masked by employing the techniques mentioned above. Furthermore, the sound and string vibrations produced by a key press are in good correspondence with the velocity with which the hammer hits a string [33]. If a keyboard is equipped with sensors complying with the MIDI protocol, then such map is encoded for each key and made available as digital messages. Together, these properties allow the experimenter to (i) record the vibratory response of the keyboard to measurable key actions; (ii) create a database of reproducible action–response relationships; (iii) make use of those data in experiments where pianists perform simple tasks on the keyboard, such as hitting one or few keys.

Our interest in the piano keyboard is not only motivated by its relatively easy experimental control: As mentioned above, its tactile feedback measured in open loop was found hardly above the standard vibrotactile sensitivity thresholds [4]. Did this evidence set an end point to the perception of piano keyboard vibrations? This chapter discusses and compares the results of two previously reported experiments on vibrotactile perception in active tasks: The first one conducted in a controlled setting and the other in an ecological, musical setting. The goal was twofold: (i) to assess how finger pressing (similar to a key-press task) affected vibrotactile detection thresholds and (ii) to investigate whether pianists perceive keyboard vibrations while playing.

Somewhat surprisingly, in Experiment 1 we found sensitivity thresholds much lower than those previously reported for passive tasks. Experiment 2 demonstrated that pianists do perceive keyboard vibration, with detection rates highest at the lower octaves and gradually decreasing toward higher pitches. Importantly, vibrations at the piano keyboard were also measured with an accelerometer for the conditions used in the experiment: While their intensity was generally lower than the standard thresholds for passive touch, conversely a comparison with the thresholds obtained in Experiment 1 provided a solid explanation to how pianists detected vibrations across the keyboard.

These findings suggest that studies on active touch psychophysics are required to better understand the relevance of haptic cues in musical performance and, consequently, to inform the development of future haptic musical interfaces.

2 Experiment 1: Vibrotactile Sensitivity Thresholds Under Active Touch Conditions

In this experiment, vibrotactile perceptual thresholds at the finger were measured for several levels of pressing force actively exerted against a flat rigid surface [43]. Vibration of either sinusoidal or broadband nature and of varying intensity was provided in return. The act of pressing a finger is indeed a gesture found while performing on many musical instruments (e.g., keyboard, reed, and string instruments) and therefore represents a case study of wide interest for musical haptics. Based on the results reported by several previous studies [9, 10, 28, 34, 36], we expected perceptual thresholds to be influenced by the strength of the pressing force.

2.1 Setup

A self-designed tabletop device called the Touch-Box was utilized to measure the applied normal force and area of contact of a finger pressing its top surface and to provide vibrotactile stimuli in return. Technical details on the device are given in Sect. 13.3.1. The Touch-Box was placed on a thick layer of stiff rubber, and sound emissions were masked by noise played back through headphones. To minimize variability of hand posture, an arm rest was used.

The experiment made use of two vibrotactile stimuli, implementing two different conditions: Band-passed white noise with 48 dB/octave cutoffs at 50 and 500 Hz and a sine wave at 250 Hz. Both stimuli focus around the range of maximal vibrotactile sensitivity (200–300 Hz [55]). During the experiment, stimulus amplitude was varied in fixed steps according to a staircase procedure (see Sect. 4.2.2). Stimulus level was calculated as the RMS value of the acceleration signal, accounting for the power of vibration acceleration averaged across the stimulation time.

Pressing force was a within-subject condition with three target levels, covering a range from light touch to hard press, while still being comfortable for participants [13], as well as compatible with forces found in instrumental practice [4]. In what follows, the three force levels are referred to as Low, Mid, and High, which correspond, respectively, to 1.9, 8, and \(15\,\mathrm {N}\), with a tolerance of \(\pm 1.5\,\mathrm {N}\).

2.2 Procedure

Twenty-seven subjects participated in the sinusoidal condition, and seventeen in the noise condition. They were 19–39-year old (\(\mathrm {mean}=26\), \(\mathrm {SD}=4.5\)), and half of them were music students. The experiment lasted between 35 and 60 min, depending on the participants’ performance, and a 1-minute break was allowed every 5 min to prevent fatigue.

Perceptual thresholds were measured using a one-up-two-down staircase algorithm with fixed step size (\(2\,\mathrm {dB}\)Footnote 1) and eight reversals, and a two-alternative forced choice (2AFC) procedure. The method targets the stimulus level corresponding to a correct detection rate of 70.7% [35], estimated as the mean of the last six reversals of the up-down algorithm.

Three staircases were implemented, each corresponding to a target force level, which were presented in interleaved and randomized fashion. Participants were instructed to use their dominant index finger throughout the experiment. A trial consisted of two subsequent finger presses, with vibration randomly assigned to only one of them. The participants’ task was to identify which press contained the vibration stimulus. Before the observation interval began, a LCD screen turning green signaled the stable reaching of the requested force level.

2.3 Results

As shown in Fig. 4.1, at each pressing force level thresholds for sinusoidal vibration were lower than for noise. For both vibration conditions, higher thresholds (i.e., worse detection performance) were obtained at the Low force condition, while at the other two force levels the thresholds were generally lower. The lowest mean threshold (\(68.5\,\mathrm {dB}\) RMS acceleration) was measured at the High force condition with sinusoidal vibration, and the highest at the Low force condition with noise vibration (\(83.1\,\mathrm {dB}\))—thus thresholds varied over a wide range across conditions. Individual differences were also large: The lowest and highest individual thresholds differ typically by about \(20\,\mathrm {dB}\) in each condition.

Fig. 4.1
figure 1

Figure reprinted from [43]

Thresholds measured at three pressing force levels, for sinusoidal and noise vibrations. Error bars represent the standard error of the mean.

Full size image

Perceptual thresholds were analyzed by means of a mixed ANOVA. A significant main effect was found for type of vibration (\(F_{1, 41}=14.64\), \(p<0.001\), generalized \(\eta ^2=0.23\)) and force level (\(F_{2, 82}=137.5\), \(p<0.0001\), \(\eta ^2=0.35\)), while the main effect of musical experience was not significant. Post hoc pairwise comparisons with Bonferroni correction (sphericity assumption was not violated in the within-subject force level factor) indicated that the Low force condition differed from both the Mid and High force conditions, for both vibration types (\(t(82)>8.85\), \(p<0.0001\) for all comparisons). For noise vibration, the difference between Mid and High force conditions was significant (\(t(82)=-3.17 \), \(p=0.02\)), but the respective contrast for sinusoidal vibration was not (\(t(82)=1.64\), \(p>0.05\)). The difference between sinusoidal and noise vibrations was significant for the Low (\(t(57.44)=4.37\), \(p<0.001\)) and High (\(t(57.44)=4.29 \), \(p<0.001\)) force conditions, but not for the Mid force (\(t(57.44)= 1.85\), \(p>0.05\)).

2.4 Discussion

Vibrotactile perceptual thresholds were found in the range 68.5–83.1 dB RMS acceleration—values that are considerably lower than what generally reported in the literature. Maeda and Griffin [36] compared acceleration thresholds from various studies addressing passive touch, finding that most of them are in the range 105–115 dB for sinusoidal stimuli ranging from 100 to 250 Hz. The lowest reported acceleration thresholds are 97–98.5 dB, for contact areas (probe size) ranging from 53 to \(176.7\,\mathrm {mm^2}\) [1, 2, 15]. It is worth noticing that the widely accepted results by Verrillo [55] report lowest displacement thresholds of approximately \(-20\,\mathrm {dB}\) (re \(10^{-6}\,\mathrm {m}\)) at 250 Hz, equivalent to about 105 dB RMS acceleration.Footnote 2

The main result of the present experiment is that vibrotactile sensitivity depends on the applied pressing force. Thresholds were highest at the Low force condition and decreased significantly at both Mid and High force levels. In good accordance with what reported in a preliminary study [44], for noise vibration the lowest threshold was obtained at the Mid force condition, while at the Low and High conditions thresholds were higher, resulting in a U-shaped threshold contour with respect to the applied force. However, as shown in Sect. 13.3.1.4, the spectral centroid of the noise vibration generally shifted toward \(300\,\mathrm {Hz}\) and higher frequencies for the Mid and High force conditions. Therefore, we suggest that the U-shape of the threshold-force curve might be partially due to the response of the Pacinian channel, which shows a U-shaped contour over the frequency range 40–800 Hz with maximum sensitivity in the 200–300 Hz range [8]. Conversely, for sinusoidal vibrations at \(250\,\mathrm {Hz}\), mean \(\mathrm {dB}\) thresholds decreased roughly logarithmically for increasing pressing forces (see Fig. 4.1). This simpler trend may be due to the more consistent behavior of our system when reproducing simpler sinusoidal vibrations (see Sect. 13.3.1.4). An improved version of the Touch-Box would be needed to test whether a similar trend can be found when noise stimuli are reproduced more linearly for varying pressing forces.

Further studies are needed to precisely assess how vibratory thresholds might be affected by passive forces of strength equivalent to the active forces used in the present study. However, since the Low condition in our experiment was already satisfied by applying light pressing force (the measured mean is about \(1.49\,\mathrm {N}\)), it may be compared to studies addressing passive static forces. Craig and Sherrick [10] found that increasing static force on the contactor produces an increase in vibrotactile magnitude. They considered vibration bursts at 20, 80, and \(250\,\mathrm {Hz}\) lasting \(1240\,\mathrm {ms}\), contact areas up to \(66.3\,\mathrm {mm}^2\), and static forces of about 0.12 and \(1.2\,\mathrm {N}\). Harada and Griffin [28] used a contact area of \(38.5\,\mathrm {mm^2}\) and found that forces in the range 1–3 N led to significant lowering of thresholds by 2–6 dB RMS at 125, 250, and \(500\,\mathrm {Hz}\). The lowest thresholds reported are however around \(100\,\mathrm {dB}\) RMS acceleration. On the other hand, Brisben et al. [9] reported that passive static contact forces from 0.05 to \(1.0\,\mathrm {N}\) did not have an effect on thresholds. However, with only four participants, the statistics of those results are not robust. Nevertheless, the authors suggested that extending these investigations to higher forces, as found in everyday life, would be important. They also hypothesized that increasing the force beyond 1–2 N could lower thresholds by better coupling of vibrating surfaces to bones and tendons, which could result in more effective vibration transmission to distant Pacinian corpuscles. That might also contribute to explain the generally lower thresholds that we found for higher forces. In our study, force level was found strongly correlated to contact area, resulting in larger areas for higher forces, which clearly contributed to further lowering perceptual thresholds [43].

Only a few related studies are found in the literature dealing with non-sinusoidal stimuli. Gescheider et al. [22] studied difference limens for the detection of changes in vibration amplitude, with either sinusoidal stimuli at 25 or \(250\,\mathrm {Hz}\) or narrowband noise with spectrum centered at \(175\,\mathrm {Hz}\) and \(24\,\mathrm {dB}\)/octave falloff at 150 and \(200\,\mathrm {Hz}\) (contact area \(2.9\,\mathrm {cm}^2\)). They found that the nature of the stimuli had no effect on difference limens.

Wyse et al. [61] conducted a study with hearing-impaired participants and found that, for complex stimuli and whole hand contact (area of about 50–80\(\,\mathrm {cm}^2\)), the threshold at 250 Hz was \(80\,\mathrm {dB}\) RMS acceleration, i.e., comparable with our results, especially in the Low force condition. In that study, it is hypothesized that the temporal dynamics of spectrally complex vibration might play a key role in detecting vibrotactile stimulation. In our case, however, the stimuli had no temporal dynamics. Sinusoidal stimuli resulted in lower RMS acceleration thresholds as compared to noise vibration. This may be explained intuitively by considering that equivalent RMS acceleration values for sinusoidal and noise stimuli actually result in a similar amount of vibration power being concentrated at \(250\,\mathrm {Hz}\) (a frequency characterized by peak tactile sensitivity [55]), or spread across the 50–500 Hz band, respectively. This explanation is supported by the findings by Young et al. [64], who reported lower thresholds produced by sinusoidal stimuli than spectrally more complex signals (square and ramp waves).

The Pacinian channel, targeted by this study, is capable of spatial summation. Previous studies [21, 55] showed that for contact areas between 2 and \(510\,\mathrm {mm}^2\) at the thenar eminence of the hand, and for frequencies in the 40–800 Hz range, displacement thresholds decrease by approximately \(3\,\mathrm {dB}\) with every doubling of the area. Intuitively, a reason for that is that the number of stimulated skin receptors increases with larger contact areas. In the present experiment, the interactive nature of the task resulted in high variability of the contact area [43]. The mean contact areas measured in the experiment were in the range 103–175\(\,\mathrm {mm^2}\), contributing to explaining the reported enhanced sensitivity.

The Pacinian channel is also sensitive to temporal summation, which lowers sensitivity thresholds and enhances sensation magnitude [21]. Verrillo [53] found that thresholds decrease for stimuli at \(250\,\mathrm {Hz}\) for increasing duration up to about \(1\,\mathrm {s}\), when delivered through a \(2.9\,\mathrm {cm}^2\) contactor to the thenar eminence of the hand. Gescheider and Joelson [23] examined temporal summation with stimulus intensities ranging from the threshold to \(40\,\mathrm {dB}\) above it: For 80 and \(200\,\mathrm {Hz}\) stimuli, peak displacement thresholds were lowered by up to about \(8\,\mathrm {dB}\) for duration increasing from 30 to \(1000\,\mathrm {ms}\). The present study made use of stimuli lasting \(1.5\,\mathrm {s}\), which likely contributed to enhancing vibrotactile sensitivity.

Large inter-individual differences in sensitivity were found in our experiment, which we could not fully explain by contact area or age. However, this observation is in accordance with other studies [1, 29, 36, 41]. Sources for large variations in sensitivity may be many. While exposure to vibration is a known occupational health issue and can cause acute impairment of tactile sensitivity [28], experience in conditions similar to the present experiment seemed a possible advantage. Therefore, we further analyzed the performance of musician participants, who are often exposed to vibrations when performing on their instruments: Indeed, musicians’ mean threshold in the Low force condition was about \(3\,\mathrm {dB}\) lower than non-musicians’, but there was no significant difference at the other force levels. Overall, enhanced sensitivity in musicians—previously observed by other authors [14, 45, 65]—could not be confirmed.

By considering actively applied forces and unconstrained contact of the finger pad, the present study adopted a somewhat more ecological approach [24] as compared to the studies mentioned above. An analogous approach was adopted by Brisben et al. [9], who studied vibrotactile thresholds in an active task that required participants to grab a vibrating cylinder. While the exerted forces were not measured, in accordance with our results much lower thresholds were reported than in the most previous literature: At 150 and \(200\,\mathrm {Hz}\), the average displacement threshold was \(0.03\,{\upmu }\)m peak (down to \(0.01\,{\upmu }\)m in some subjects), which is equivalent to RMS acceleration values of \(85.5\,\mathrm {dB}\) at \(150\,\mathrm {Hz}\), and \(90.5\,\mathrm {dB}\) at \(200\,\mathrm {Hz}\). The authors suggested that such low figures could be due to the multiple stimulation areas on the hand involved in grabbing the vibrating cylinder, the longitudinal direction of vibration, and the force exerted by the participants. A few studies report that active movement results in lower sensitivity thresholds [63] or better percept possibly due to the involvement of planning and additional cognitive load as compared to the passive case [52].

Despite its partially ecological setting, this experiment kept control over the generation of sinusoidal and noise vibrations, with focus on the region of maximal human vibrotactile sensitivity (200–300 Hz). Vibratory cues at the piano keyboard, however similar in form to the respective tones, are more complex than either of the conditions in Experiment 1 and are likely to be perceived differently depending on the type of touch and the number of depressed keys. The following experiment tested first vibration detection in a piano-playing task, and second whether active touch sensitivity threshold curves of Experiment 1 could predict the measured results.

3 Experiment 2: Vibration Detection at the Piano Keyboard During Performance

A second experiment investigated vibrotactile sensitivity in a musical setting [19]. Specifically, the goal was to measure the ability of pianists to detect vibration at the keyboard while playing. Vibration detection was measured for single and multiple tones of varying pitch, duration, and dynamics.

3.1 Setup

The experiment was performed at two separate laboratories using similar setups, centered around two Yamaha Disklavier pianos: A grand model DC3 M4 and an upright model DU1A with control unit DKC-850. The Disklaviers are MIDI-compliant acoustic pianos equipped with sensors for recording performances and electromechanical motors for playback. They can be switched from normal operation to a “silent mode.” In the latter modality, the hammers do not hit the strings and therefore the instruments neither resound nor vibrate, while their MIDI features and other mechanical operations are left unaltered. The two setups are shown in Fig. 4.2.

Fig. 4.2
figure 2

Figure adapted from [19]

The two Disklavier setups used in the experiment. Left: Yamaha DC3 M4 grand piano. Right: Yamaha DU1A upright piano.

Full size image

During the experiment, the normal and silent modes were switched back and forth across trials, letting participants receive respectively either natural or no vibrations from the keys. In both configurations, participants were exposed to the same auditory feedback produced by a physical modeling piano synthesizer (Modartt Pianoteq), set to simulate either a grand or an upright piano, and driven in real time by MIDI data sent by the Disklaviers. The synthesized sound was reproduced through Sennheiser HDA-200 isolating reference headphones (grand piano) or Shure SE425 earphones (upright piano). In the latter case, 3M Peltor X5A earmuffs were worn on top of the earphones for additional isolation. Preliminary testing confirmed that through these setups the Disklaviers’ operating modes (normal or silent) were indistinguishable while listening to the piano synthesizer from the performer’s seat position, meaning that any acoustic sound coming from the pianos in normal mode was fully masked.

The loudness and dynamic response of the piano synthesizer were preliminary calibrated to match those of the corresponding Disklavier model in use (details are given in Sect. 13.3.2).

Participants could sense the instrument’s vibration only through their fingers on the keyboard. Other sources of vibration were excluded: The pedals were made inaccessible, while the stool, the player’s feet, and the piano were isolated from the floor by various means [17]. Vibration measurements confirmed that, as a result of the mechanical insulation, playing the piano did not cause vibrations at the player’s seat exceeding the noise floor in the room.

The experiment was conducted under human control, with the help of software developed in the Pure Data environment, which was used to: (i) read computer-generated playlists describing the experimental trials; (ii) set the Disklavier’s playing mode accordingly; (iii) check if the requested tasks were executed correctly; (iv) record the participants’ answers.

3.2 Procedure

Sensitivity was measured at six A tones of different pitch ranging from A0 to A5, chosen after a pilot study [17], reporting a significant drop in detection above A5. Tone duration was either “long” (8 metronome beats at 120 BPM) or “short” (2 beats), and dynamics either “loud” (mf to ff, corresponding to MIDI key velocities in the range 72–108) or “soft” (p to mp, key velocities 36–54). In addition to single tones, participants were requested to play three-tone clusters around D4 and D5.

The experiment consisted of two parts: In part A, participants played long and loud single tones; in part B, tone dynamics and duration were modified so as to make the detection task potentially harder in the low range, where vibrations should be most easily perceived [17]. Additionally, by extending the contact area, the note clusters were expected to facilitate detection in the high range, where sensitivity should be low [17]. The conditions are summarized in Table 4.1.

The experiment followed a 2AFC (yes/no) procedure, which required participants to report whether they had detected vibrations during a trial or not. Each condition was repeated eight times in normal mode and eight times in silent mode, in randomized order. However, part A was performed before part B.

Table 4.1 Factors and conditions in the piano experiment
Full size table

Participants were instructed to use their index fingers for single keys or fingers 2-3-4 for chords and to play pitches lower than the middle C with their left hand and the rest with the right hand.

Fourteen piano students participated in the upright piano condition, and fourteen in the grand piano condition. Their average age was 27 years and they had in average 15 years of training, mainly on the acoustic piano.

3.3 Results

Sensitivity index \(d'\), as defined in signal detection theory [26], was computed for each subject and condition as follows:

$$\begin{aligned} d'= Z(\text {hits}) - Z(\text {false alarms}), \end{aligned}$$

where Z(p) is the inverse of the Gaussian cumulative distribution function, hits is the proportion of “yes” responses with vibrations present, and false alarms is the proportion of “yes” responses with vibrations absent. Thus, a proportion of correct responses \(p(c)=0.69\) corresponds to \(d'=1\) and chance performance \(p(c)=0.50\) to \(d'=0\). Perfect proportions 1 and 0 would result in infinite \(d'\) and were therefore corrected by (\(1-1/16\)) and (1 / 16), respectively [26].

Fig. 4.3
figure 3

Figures reprinted from [19]

Sensitivity \(d'\) in part A (top) or parts A and B (bottom). Error bars represent the standard error of the mean  [40]. Chance performance (\(d'=0\)) is represented by the dashed line.

Full size image

Results of part A are presented at the top of Fig. 4.3: Sensitivity was highest in the lower range and decreased toward higher pitches. At A4 (\(440\,\mathrm {Hz}\)), vibrations were still detected with mean \(d'=0.84\), while at D5 (\(587\,\mathrm {Hz}\)) and A5, performance dropped to chance level. A mixed ANOVA indicated a significant main effect of pitch (\(F(6,156)=26.98\), \(p<0.001\)). The results for the upright and the grand piano did not differ significantly, nor was there a significant interaction of pitch and piano type. The Mauchly test showed that sphericity had not been violated. The results were collapsed over upright and grand pianos, and a trend analysis was conducted. A linear trend was significant (\(t(156)=-12.3\), \(p<0.0001\)), indicating that as pitch increases, sensitivity to vibrations decreases. Results from parts A and B are presented together at the bottom of Fig. 4.3, showing small differences in mean sensitivity between normal, soft, and short conditions. However, none of the contrasts between long and short duration or loud and soft dynamics at A0 or A1 was significant. The difference was more notable between clusters and single notes: For the cluster CDE4, sensitivity was significantly higher than for the isolated note D4 (\(t(294)=5.96\), \(p<0.0001\)), whereas the much smaller difference between D5 and the cluster CDE5 was not significant. Even considering the possible effect of learning between part A and B (average sensitivity at pitches A0 and A1 was 0.23 higher in part B), the result suggests that at D4, playing a cluster of notes facilitates vibration detection.

3.4 Vibration Characterization

In order to gain further insight into the results, vibration signals at the keyboard were measured on both the grand and upright Disklaviers.

An in-depth description of the measurements and related issues is given in Sect. 13.3.2.2. For convenience, only essential details are reported here. Vibration signals were acquired for different MIDI velocities at each of the 88 keys of the Disklavier pianos via a measurement accelerometer and recorded as audio signals. A digital audio sequencer software was used to record vibration signals, while reproducing MIDI tracks that played back each single key of the Disklaviers. Additional MIDI tracks were used to play CDE4 and CDE5 clusters, while vibration was recorded with the accelerometer attached to the respective C, D, and E keys in sequence. The MIDI velocities were chosen to cover the entire dynamics reproducible by the Disklaviers’ motors.

Acceleration signals had a large onset in the attack, corresponding to the initial fly of the keys followed by their impact against the keybed. Figure 4.4 shows a typical attack, recorded from the grand Disklavier playing the A2 note at MIDI velocity 12. These onsets, appearing in the first 200–250 ms, are not related to the vibratory response of the keys and were therefore manually removed from the samples.

Fig. 4.4
figure 4

Figure reprinted from [19]

Attack of the acceleration signal recorded for note A2, MIDI velocity 12, grand Disklavier.

Full size image
Fig. 4.5
figure 5

Figure reprinted from [19]

Vibration spectrum of A0 played with ff dynamics (MIDI velocity 111) on the upright Disklavier, represented as magnitude acceleration in \(\mathrm {dB}\). The vertical dotted line shows the nominal fundamental frequency \(\mathrm {f0}=27.5\,\mathrm {Hz}\). The dashed curve represents vibrotactile acceleration thresholds at the fingertip adapted from Verrillo [55].

Full size image

Acceleration values in \(\mathrm {m/s}^2\) were computed from the acquired signals by making use of the nominal sensitivity parameters of the audio interface and the accelerometer. Similarly to what was done by Askenfelt and Jansson [4], the spectra of the resulting acceleration signals were compared to Verrillo’s reference vibrotactile sensitivity curve [55]. Note that this curve reports sensitivity as the smallest, frequency-dependent displacement A(f) (in meters) of a sinusoidal stimulus \(s(t)=A(f) \sin (2\pi ft)\) that is detected at the fingertip. Therefore, a corresponding acceleration curve was computed from the original displacement curve in order to compare with our acceleration signals. Thanks to the sinusoidal nature of the stimuli employed by Verrillo, the corresponding acceleration signal could be found analytically as \(\ddot{s}(t)=-A(f)\cdot (2\pi f)^2\sin (2\pi ft)\). Consequently, the acceleration threshold curve \(A(f)\cdot (2\pi f)^2\) was used for comparison to our signals. Confirming the results by Askenfelt and Jansson [4], no spectral peaks were found to exceed the acceleration threshold curve, even for notes played with high dynamics. To exemplify this, Fig. 4.5 shows the spectrum of the highest dynamics of the note that participants detected with the highest sensitivity (part A), i.e., A0 played at MIDI velocity 111, along with the threshold curve.

Since Verrillo’s thresholds cannot explain the results of Experiment 2, RMS acceleration values were computed in place of spectral peak amplitudes, in analogy with Experiment 1. Vibration signals were first processed with a specifically designed low-pass filter to shape stimuli according to human vibrotactile band [19]. RMS values in \(\mathrm {dB}\) were then extracted from the filtered signals over time windows equal to the lengths of the stimuli, that is \(1\,\mathrm {s}\) for short and \(4\,\mathrm {s}\) for long trials. Figure 4.6 shows the resulting RMS values for parts A and B, respectively, together with the RMS thresholds of vibration reported in Experiment 1. A comparison of the RMS acceleration values and perceptual thresholds for noise shown in these figures against the sensitivity curves of Fig. 4.3 suggests that RMS values of broadband stimuli have more potential to explain the results of Experiment 2.

Fig. 4.6
figure 6

Figure adapted from [19]

RMS acceleration values of keys played as in part A (top) or parts A and B (bottom). The horizontal lines represent (min/max) vibrotactile thresholds as measured in Experiment 1 for noise and sinusoidal stimuli over a range of active pressing forces.

Full size image

3.5 Discussion

The results presented in the previous section show that sensitivity to key vibrations is highest in the lowest range and decreases toward higher pitches. Vibrations are clearly detected in many cases where the vibration acceleration signals hardly reached typical thresholds found in the literature for sinusoidal stimuli.

The literature on the detection of complex stimuli provides support to our results, although it does not explain them completely. As already discussed in Sect. 4.2.4, Wyse et al. [61] report RMS acceleration threshold values at \(250\,\mathrm {Hz}\) corresponding to \(80\,\mathrm {dB}\), a value compatible with our results. However, the characteristics of those stimuli may have occasionally produced significant energy at lower frequencies, causing the thresholds to lower once they were presented to the whole hand.

The pianist receives the initial transient when the hammer hits the string; then, the vibration energy promptly decreases and its partials fade each with its own decay curve. The initial peak may produce an enhancement effect similar to those measured by Verrillo and Gescheider limited to sinusoids [56] and hence contribute to sensitivity.

As discussed earlier, the P-channel is sensitive to the signal energy, while is not able to recognize complex waveforms. Loudness summation instead occurs when vibration stimulates both the Pacinian and non-Pacinian (NP) channels, lowering the thresholds accordingly [7, 37, 56]. In our experiment, summation effects were likely to occur when the A0 key and, possibly, the A1 key were pressed. From A3 on, only the P-channel became responsible for vibration perception. Figure 4.3 seems to confirm these conclusions, since they show a pronounced drop in sensitivity between A1 and A3 in both parts of Experiment 2. As Fig. 4.6 demonstrates, this drop is only partially motivated by a proportional attenuation of the vibration energy in the grand piano, while it is not motivated at all in the upright piano. Hence, it is reasonable to conclude that the NP-channel played a perceptual role until A3. Beyond that pitch, loudness summation effects ceased.

In analogy with Experiment 1, the results of this experiment also suggest the occurrence of spatial summation effects [10] when a cluster of notes, whose fundamentals overlap with the tactile band, is played instead of single notes. As Fig. 4.3 (bottom panel) shows, playing the cluster in the fourth octave boosted the detection in that octave, whereas the same effect did not occur in the fifth octave. Unlike Experiment 1, this summation originates from multifinger interaction rather than varying contact areas in single-finger interaction. This evidence opens an interesting question about the interaction of complex vibrations reaching the fingers simultaneously. Measurements of cutaneous vibration propagation patterns in the hand resulting from finger tapping show, however, an increase in both intensity and propagation distance with the number of fingers involved [49], which may partially explain the increased sensitivity we observed.

Unlike Experiment 1, where uni-modal tactile stimuli were used, here we employed bimodal audio-tactile stimuli. Therefore, the possibility of cross-modal amplification effects needs to be shortly discussed, even though Experiment 2 did not investigate this aspect. As discussed earlier, previous studies on cross-modal integration effects [46, 58] support the concrete possibility that an audible piano tone, whose vibratory components are a subset of the auditory components, helps detect a tactile signal near threshold. Although in our case the sound came from a synthesizer, both the auditory and tactile signals shared the same fundamental frequency of the piano tone, and furthermore the first partials were close to each other, respecting the hypothesis of proximity in frequency investigated by Wilson [58]. We did not test a condition in which subjects played the piano in normal mode in the absence of auditory feedback, or using sound uncorrelated with vibration (e.g., white noise). Although that may provide significant data about the effective contribution of auditory cues to vibration detection on the piano, a different experimental setup should be devised. Other cross-modal effects that may have instead contributed to impair the detection [62] should be considered as minor with respect to the spectral compatibility and temporal synchronization of the audio-tactile stimulus occurring when a piano key was pressed.

Yet another relevant difference with Experiment 1 is that in this case the pressing forces exerted by pianists were unknown and most likely not constant throughout a single trial. The maximum and minimum sensitivity thresholds lines in Fig. 4.6, which report the results of Sect. 4.2.3, correspond to constant pressing forces of 1.9 and \(8\,\mathrm {N}\) for noise vibration, and 1.9 and \(15\,\mathrm {N}\) for \(250\,\mathrm {Hz}\) sinusoidal vibration. These force values occur when piano keys are hit at dynamics between pp and f, with negligible difference between struck and pressed touch style [20, 33]. Conversely, ff dynamics require stronger forces up to \(50\,\mathrm {N}\) [4]. In Experiment 2, it seems reasonable to assume that pianists initially pressed the key according to the dynamics required by the trial and then, once the key had reached the keybed, accommodated the finger force on a comfortable value while attending the detection process. If our participants adapted finger forces toward the range mentioned above, then their performance in this experiment would fall in between the results for sinusoidal and noise stimuli in Experiment 1. Experiment 1 additionally found that, when using low finger force, musicians on average exhibit slightly better tactile acuity than non-musicians. Even if this difference was not significant, our participants could have reduced the finger force only after starting a trial that required loud dynamics, while leaving the force substantially unvaried during the entire task in the other cases. This behavior seems indeed quite natural.

The hypothesis that vibrotactile sensitivity to RMS acceleration falls in between the thresholds for \(250\,\mathrm {Hz}\) sine wave and filtered noise is coherent with the temporal and spectral characteristics of the stimuli: Right after its initial transient, a piano tone closely resembles a decaying noisy sinusoid. For instance, it can be simulated by employing several hundreds of damped oscillators whose outputs are subsequently filtered using a high-order transfer characteristic [5]. A remaining question is whether the RMS acceleration values of filtered noise plotted in Fig. 4.6 explain our thresholds sufficiently, or if there is a need to discuss them further. Other elements in favor of further discussion are the mentioned potential existence of a cross-modal amplification and evidences of superior tactile acuity in musicians [65].

4 Conclusions

We have given an introduction to the role of active touch in musical haptic research. A closed loop between musicians and their instrument during performance poses a major challenge to experimental setups: While playing, musicians generate themselves the vibrotactile feedback and are at the same time influenced by the produced sound. To discuss the possible links between music performance tasks and basic active touch psychophysics, we presented two experiments, one in a controlled and one in an ecological setting, showing evidence that pianists perceive keyboard vibrations with sensitivity values resembling those obtained under controlled active touch conditions. Overall, the results presented here suggest that research on active touch in musical performance may prove precious to understand the role, mechanisms, and prospective applications of active touch perception also outside the musical context. An example application that seems at immediate reach of current tactile interfaces is to create illusory effects of loudness change by varying the intensity of vibratory feedback [39, 42].

Although interesting and necessary, our results represent only a premise for further research activities aimed at precisely understanding the role of tactile feedback during piano playing. Exploratory experiments have already been performed in an attempt to understand whether changes in the “timbre” of tactile feedback may determine equivalent auditory sensations. Some results in this regard are presented in Sect. 5.3.2.2. If confirmed, after excluding the influence of non-airborne sonic cues on auditory perception, such results would imply the ability of the tactile and auditory systems to interact so as to form a wider, multimodal notion of musical timbre, for which some partial evidence has been found in musicians [59] and non-musicians [47]. Several questions related to the role of tactile feedback in musical performance remain open. For instance, feedback from percussion instruments is likely to define strong patterns of skin vibration extending far beyond the interaction point. The propagation of vibration across the skin has been recent object of research having potentially interesting haptic applications outside the musical context [49]. It cannot be excluded that percussionists control their playing by testing specific wide-area tactile patterns they learned, and then retained in the somatosensory memory after years of practice with their instrument: Some sense of unnatural interaction with the instrument otherwise should not be experienced by drummers and percussionists when they play rubber pads and other digital interfaces. Furthermore, while it is not precisely known how wind instrument players make use of the vibrations transmitted by the mouthpiece, digital wind controllers like the Yamaha WX series never achieved wide popularity, possibly also due to their unnatural haptic feedback.