Tactile discrimination of material properties: application to virtual buttons for professional appliances

Abstract

An experiment is described that tested the possibility to classify wooden, plastic, and metallic objects based on reproduced auditory and vibrotactile stimuli. The results show that recognition rates are considerably above chance level with either unimodal auditory or vibrotactile feedback. Supported by those findings, the possibility to render virtual buttons for professional appliances with different tactile properties was tested. To this end, a touchscreen device was provided with various types of vibrotactile feedback in response to the sensed pressing force and location of a finger. Different virtual buttons designs were tested by user panels who performed a subjective evaluation on perceived tactile properties and materials. In a first implementation, virtual buttons were designed reproducing the vibration recordings of real materials used in the classification experiment: mainly due to hardware limitations of our prototype and the consequent impossibility to render complex vibratory signals, this approach did not prove successful. A second implementation was then optimized for the device capabilities, moreover introducing surface compliance effects and button release cues: the new design led to generally high quality ratings, clear discrimination of different buttons and unambiguous material classification. The lesson learned was that various material and physical properties of virtual buttons can be successfully rendered by characteristic frequency and decay cues if correctly reproduced by the device.

Introduction

Even though everyday human interaction with objects and events is mostly multisensory  [37], sight is often fundamental. There are however situations in which one can rely on auditory and/or haptic cues only, for instance when a user is involved in multiple activities, or when an interface is visually occluded.

As a specific case study, the classification of materials is usually mainly based on visual cues  [35], however it may also rely on touch and/or audition. For instance, this happens as a consequence of tactile exploration or other excitation (e.g., by tapping) of an object’s natural resonances  [31]. Several studies are found in the literature which investigated the human ability to identify materials via the auditory or haptic modalities. Audition has been tested using synthetic (simplified) stimuli  [12, 25], revealing good performance in material classification for virtual sounding objects defined by the characteristics of their modes of resonance (frequency, amplitude and decay time). Another study  [4] reported that, when subjects were asked to grasp an object made of a given material, congruence between visual and auditory cues (contact sounds) improved the identification performance, while incongruent sounds gave rise to perceptual interference. The capabilities of the haptic channel have been tested by reproducing feedback related to materials resonances on haptic displays. A few experiments revealed successful material identification with vibrations and force feedback in response to a tapping action  [14, 34]. Several studies approached the perception of surface and material properties by focusing on multisensory integration of auditory and haptic cues, as well as cross-modal effects  [3, 22, 36]. For instance, the parchment-skin illusion experiment showed how acoustic information affects the perceived roughness during hand rubbing  [18], while another study demonstrated differences in the perceived stiffness through audition or touch  [27].

With regard to applications in the context of virtual environments of the perceptual research outlined above, the simulation of material properties via multimodal interaction is a hot topic. While auditory-based simulations can rely on affordable, high-fidelity and well-spread technology, the haptic reproduction of properties such as roughness or compliance is far from trivial. Due to the complexity of the human somatosensory system, different haptic technologies generally focus on rendering characteristics related to specific interactions such as pressing  [10] or sliding  [8, 17]. However, vibrotactile cues have been studied for conveying illusory intra-modal effects of depth, compliance, roughness or indentation  [20, 24, 30]. An application scenario for such researches that is of high relevance for human-computer interaction is offered by virtual buttons. Several studies addressed the design and evaluation of virtual buttons  [16, 19, 21], as well as the possibility to render various characteristics of physical buttons  [20, 28, 33]. In particular, Lee et al.   [26] performed different experiments to compare the performance of hard (physical) and soft (capacitive or resistive) buttons: Even though the scores related to the two types of buttons were comparable, users evaluated button activations by pressure (i.e., resistive) to be more reliable than by contact (i.e., capacitive).

This paper first contextualizes part of a previously reported experiment  [6, 7], which studied the classification of various materials based on auditory and vibrotactile cues of impact events. The envisioned application of the experiment was the enrichment of touchscreens on professional appliances with robust vibrotactile feedback so as to render specific materials and other tactile characteristics.

Supported by the experimental results, in the second part of the paper a prototype touchscreen device implementing several virtual buttons that render various tactile properties and materials underwent subjective evaluations by two user panels in separate case studies. Due to the limited bandwidth of the hardware, the mere reproduction of the vibrotactile stimuli used in the classification experiment resulted generally ineffective, as reported in the first case study. Hence, the last part of the paper describes a second case study making use of an alternative virtual buttons design carefully optimized for the hardware, which was conversely judged positively.

Material classification experiment

Method

The experiment assessed and compared the robustness of material classification based on reproduced unimodal (auditory or vibrotactile) or bimodal (audio-tactile) feedback.

Setup

Fig. 1

From left to right: wood, plastic, and metal objects used to record stimuli and train participants

Auditory and vibratory stimuli were prepared from recordings of single hits of a ping-pong ball dropped from a height of 40 cm on three same-shaped objects, respectively made of fir wood, hard plastic, and steel. The objects were custom-made, and their U-shape was chosen so as to allow a hand or an accelerometer to find place underneath (see Fig. 1). A ping-pong ball was chosen for its low weight and hardness, after conducting informal comparisons against metal, rubber and wooden balls; these, mainly due to their weight, gave rise to low-frequency vibrations which are not reproducible by small, low-power vibration exciters, and were therefore discarded.Material classification on such three objects had been preliminary tested in the same conditions, giving accurate results  [6].

Fig. 2

Spectrograms of the recorded impact sounds on wood, plastic and metal

Sound was recorded 40 cm away from the point of impact using an Audio-Technica AT4050 condenser microphone (omni pattern) connected to a RME Babyface audio interface.

Fig. 3

Spectrograms of the originally recorded (left) and reproduced (right) impact vibrations acquired on the sample objects of Fig. 1

The respective vibrations were recorded by attaching a Wilcoxon 736 accelerometer to the bottom of the objects, in correspondence of the point of impact of the ball. Figures 2 and 3 (left) report the spectrograms of the recorded stimuli. The RMS power of the recorded signals was normalised within a 500-ms window, thus preventing participants from using the feedback intensity as a cue (e.g., in everyday life resonances generated by metal objects are usually stronger than those associated with wood). The prepared stimuli are made available via an open-access online repository.¹

Fig. 4

Haptic display

Auditory stimuli were played back through Beyerdynamic DT 770 PRO closed-back headphones, while vibrotactile stimuli were reproduced via a custom-made haptic display (see Fig. 4) consisting of a Dayton Audio DAEX32Q-4 vibrotactile transducer (W 55 $\times$ H 19.5 mm, weight 132.6 g) fixed on top of a borosilicate glass plate (L 250 $\times$ W 210 $\times$ H 3 mm). The plate was suspended by placing it on rubber strips held by a metal frame, while the latter was hanging on a wooden structure that kept it raised from the support table. This configuration allowed participants to touch the glass plate from below, as shown in Fig. 4. Unimodal vibrotactile stimuli were accompanied by auditory masking noise via headphones.

Fig. 5

Power spectra in the 20–1000 Hz range of the reproduced vibrations for impacts over wood, plastic and metal

For validation, the reproduced vibrations were re-recorded by attaching the accelerometer to the bottom of the glass plate right below the transducer (i.e., where finger contact would take place), and compared with the respective signals originally recorded on the three sample objects of Fig. 1. As shown in Fig. 3, while the reproduced signals are generally similar to the original ones in the range of tactile perception (up to 1000 Hz), marked harmonic distortion is introduced and lower-frequency components lose energy. Namely, due to the limited bandwidth and dynamics of the actuator, and to the inherent resonances of the glass plate, vibrotactile cues were not accurate reproductions of the original vibration signals. The overlapped power spectra of the reproduced vibrations are shown in Fig. 5.

The described setup hence resulted in high-fidelity auditory and degraded vibrotactile reproduction. The rationale for reproducing vibrotactile cues under such conditions is that they represent a worst-case scenario accounting for the working conditions of professional environments where several sources of noise may be present, and tactile cues may be distorted in various ways (e.g., due to additional vibrations produced by machinery or because of gloves worn by users). At the same time, the use of high-fidelity auditory feedback—which would be hardly effective in noisy environments—allowed to set a reference baseline for non-visual material classification.

From here on, the reproduced stimuli will be labeled according to two factors: Material (Wood, Plastic, Metal) and Modality (unimodal Auditory, unimodal Tactile, and Bimodal audio-tactile).

The masking noise delivered through headphones in the Tactile condition faded-in and and faded-out respectively about 2 s before and 2 s after the presentation of tactile stimuli, thus preventing participants from hearing sound leaking from the actuator. In the Bimodal condition, the start of auditory and tactile stimuli was time-aligned to account for the travelling time of sound waves running the distance between the participant’s head and the haptic display where the impact events were virtually taking place (1.14 ms delay on audio signals).

Participants

Twenty-seven participants (20 males, 7 females), aged between 21 and 54 (${\mathrm{M}}=29.0$; ${\mathrm{SD}}=6.8$) were invited, all reporting normal hearing and touch ability. Before the experiment, such abilities were informally tested by asking participants to close their eyes, localize a sound source nearby, and finally recognize the test objects of Fig. 1 by touch. No compensation was offered for taking part in the experiment.

Table 1 Confusion matrix for each Modality

Procedure

The experimental task was to identify a material from reproduced stimuli.

Initially participants were briefed about the experimental protocol and procedure, and then they familiarized with material cues: the experimenter hit the objects of Fig. 1 with a ping-pong ball, while participants listened to the produced sounds and felt the resulting vibrations by placing one finger-pad of their dominant hand underneath the surface. Familiarization continued until participants felt confident with all materials.

In each session, factors were crossed and each factor combination was repeated six times, resulting in 6 $\times$ 3 Material types $\times$ 3 Modality types $=$ 54 trials. Trials were organized in blocks of 18, according to Modality. The unimodal Auditory and Tactile conditions were both presented before the Bimodal one, and their order was balanced between participants. Within each block, the repetitions of each Material were presented in random order. A single session lasted about 10 minutes.

Results

The overall performance was well above chance level (33%), except for two participants who scored at chance level with both unimodal Auditory and Tactile feedback, and four who did the same in a single Modality.

Table 1 shows the average distributions of responses given in the Auditory, Tactile, and Bimodal conditions.

Highlighted in bold, diagonals report the average rates of correct response (i.e., matching stimulus and response), while the other cells report values for mismatched responses. The columns labeled ‘none’ give rates of no answer to the presented stimuli.

Unimodal feedback

In general Auditory scored better than Tactile, resulting in higher matching rates, especially concerning Wood and Plastic.

Tests on the unimodal distributions with the D’Agostino method confirmed no significant deviation from normality for all factors  [5]. Figure 6 represents the matching results for the Auditory and Tactile feedback conditions respectively as black and blue box plots.

A two-way repeated measures ANOVA was conducted to test the influence of the two independent factors Modality and Material on the proportion of correct responses. Using a Greenhouse-Geisser correction for insphericity, the effect of Material was found statistically significant with ${{F}}(1.61, 41.9)=16.3$, ${{p}}<0.001$, suggesting a significant difference between Wood (${{M}}=0.72$, ${{SD}}=0.033$), Plastic (${M}=0.58$, ${{SD}}=0.033$) and Metal (${M}=0.50$, ${{SD}}=0.04$), whereas the effect of Modality was not significant (${p}=0.09$). The interaction between the two factors was not significant either (${p}=0.563$).

Confidence Intervals 95% resulted in partial overlap between Plastic [0.51, 0.64] and Metal [0.42, 0.57], while Wood was outside their combined range [0.65, 0.78]. This fact is visible also in Table 1, reporting that Plastic-Metal and Metal-Plastic pairs (respectively, stimulus and response) were on average confused more than the other material pairs.

Bimodal feedback

Categorization was generally better in the Bimodal condition, with scores up to 12% and 20% higher than the Auditory and Tactile ones, respectively. As shown by the purple box plots in Fig. 6, the Bimodal scores distribution was not normal due to a ceiling effect.

A non-parametric Friedman test  [9] detected a significant main effect of Modality (${Q}=25.0$, ${p}<0.01$). Pairwise comparisons were performed using the Wilcoxon Rank-sum test  [15], revealing significant differences between Auditory-Bimodal (${Z}=-2.5$, ${p}=.03$ Bonferroni corrected) and Tactile-Bimodal (${Z}=-3.7$, ${p}<.01$).

Fig. 6

Box plots for all factor combinations

Discussion

When finished, all participants reported to have found the test more difficult than expected, and to feel uncertain about their results. Indeed, with three materials the probability to perform at chance level (33.3%) is fairly high, while the probability to make a correct guess in at least half of the responses (that is, 9 out of 18 trials in each block) drops to 10.8%.

Concerning auditory-based material identification, previous literature focusing on synthesized sound stimuli  [12, 25] reports successful discrimination that seems to mainly rely on decay and frequency cues. On average our results in the Auditory condition revealed worse performance, however they are related to the reproduction of sonic feedback recorded on real objects. This may depend on the spectral complexity and noisier nature of the stimuli, whereas synthetic sounds used in the literature were generated with only a few, well-defined modes. Moreover, our stimuli were affected by the coupling of the objects of Fig. 1 with the support table, which inevitably altered their characteristic resonances. In fact, as shown in Fig. 2 the decay times of the different materials are rather similar, and the Metal stimuli were the most affected, with severely damped decays. Yet, in everyday scenarios one rarely finds ideally undamped interactive objects, as they are usually handheld, embedded in bigger structures, or laying on a surface (as in our case).

Giordano et al.  [13] studied material identification using plates made of wood, steel, plexiglas and glass: they found a perfect identification of the gross categories steel-glass and wood-plexiglas, whereas materials within such categories could not be discriminated. These results may be compared to those related to Wood and Plastic in our experiment, which however were rarely mismatched. Nevertheless, the mentioned experiment addressed material categorization also involving the effect of plate size, which generally affects both pitch (lower for bigger objects) and decay time (longer for bigger objects).

Although the Tactile condition resulted in generally lower correct response rates, some participants scored even better in this modality. This made the ANOVA test on the factor Modality statistically not significant for the two unimodal conditions, suggesting that vibrotactile feedback can equivalently replace auditory feedback in material recognition tasks. This result is especially relevant since, by design, the Auditory stimuli in our experiment well preserved the nature of the original recordings (thus serving as a best-case reference), whereas the Tactile ones were degraded reproductions of the originals.

Anyhow, the average correct response rate in the Tactile condition was lower than what found in related studies  [14, 30] reporting discrimination rates up to 85%. This difference may be explained mainly by the degraded nature of the Tactile stimuli and the damped decays due to the table supporting the sample objects used in our experiment. Other relevant differences are found in the choice of materials (aluminium, rubber, wood in the mentioned experiments), the user task (active tapping) and the consequent additional kinesthetic cues that were conversely absent in our experiment.

Plastic-Metal and Metal-Plastic pairs were often confused. An explanation to this may be suggested by the overlapped power spectra of the reproduced vibration stimuli for the three materials, shown in Fig. 5: whereas Wood clearly differs from the other materials below 100 Hz, Plastic and Metal have similar energy in such region; conversely, the spectra of Plastic and Metal dramatically diverge between 300 and 600 Hz but, since tactile sensitivity progressively decreases above 300 Hz  [2], this contribution did not support their discrimination.

Overall, Bimodal feedback gave significantly better results as compared to the unimodal conditions, thus highlighting the constructive contribution of coherent multimodal stimuli. Anyhow, on average the Plastic-Metal and Metal-Plastic pairs were still confused more than other material pairs.

Design and evaluation of virtual buttons

Professional environments often present acoustic and tactile disturbances as well as visual occlusion, and therefore machines with virtual buttons providing well differentiated tactile cues may support a more effective non-visual human-machine interaction  [11].

Prototype device

Supported by the experimental results on material categorization based on tactile cues, a prototype device was designed which generates vibrotactile feedback in response to touch interactions. The device implements virtual buttons triggered by variable pressing forces in a soft-touch range (0–5 N). The main goal of the device is to render virtual buttons that are easy to discriminate based on tactile cues only. As active touch enhances the sensitivity to vibrations  [29, 32], even better discrimination performance was expected compared to the reported experiment, which was conducted in passive conditions.

Fig. 7

Schematic of the prototype device: A piezo-electric actuator (ii) is glued to the back of a 2.8 inches capacitive touchscreen (i) suspended on foam strips (iii); The touchscreen is connected to a microcontroller board (iv) that lays on a load-cell (v); A separate board hosts a driver for the piezo-electric actuator (vi). All the elements are fixed to a metal base simulating the internal panel of industrial appliances

Figure 7 shows the layout of the prototype, built using off-the-shelf components.

The device displays virtual buttons on a 2.8 inches TFT touchscreen (see Fig. 9) whose capacitive layer locates finger contact positions, while the exerted pressing force is measured using a BND-611N load-cell (0–1 kg) placed at the bottom of the structure. The load-cell is driven by a 24 bit HX711 AD converter with a sampling rate of 80 Hz. Although techniques exist for the estimation of finger force during tapping actions  [33], a more direct and accurate measure via a low-cost load-cell was preferred. Moreover, the use of a force sensor allows tracking the release phase of pressing gestures before a finger loses contact with the touchscreen surface, which would not be reliable based only on capacitive or resistive sensing.

Vibrotactile feedback is generated by a Samsung Electro Mechanics (SEMCO) PHAH353832 piezoelectric actuator (dimensions L $35\times \hbox {W}$ $3.8\times \hbox {H}$ 3.2 mm, weight 2.7 g) controlled by a Texas Instrument DRV2667 piezo driver, connected to an Arduino Mega 2560 microcontroller board via the I2C communication bus. Compared to other haptic technologies (e.g., LRA, ERM and voice coil), piezo actuators can render fast transients at different frequencies. Other benefits are their small size and low power consumption.

The piezo driver may operate in analog mode, by amplifying (up to $200\,{V}_{{pp}}$) an audio-level signal at its analog input, or use the internal digital-controlled synthesis engine to generate simple sequences of sine waves, whose parameters (frequency, amplitude, attack and decay time, and duration) can be defined.

Since the touchscreen is suspended on a foam layer, the device is slightly compliant to external pressure ($\le 1\,{{mm}}$).

Fig. 8

Frequency response of the device in the range 50–1000 Hz

The frequency response of the system was measured in the range 50–1000 Hz by attaching a Wilcoxon 736 accelerometer on top of the touchscreen (center position). As shown in Fig. 8, the device is mostly efficient around the resonance frequency of the piezo actuator (230 Hz), whereas it is substantially unable to reproduce frequencies below 100 Hz. Concerning the upper part of the tested range, artifacts are present above 700 Hz, also resulting in audible distortion.

Fig. 9

Visual appearance of the virtual buttons. For the characterization procedure, an accelerometer was placed in the middle of the touchscreen

The device displays up to four virtual buttons labeled A, B, C, D (see Fig. 9), matching the number of main functions commonly found on professional appliances (2 to 6). Their shape and size (squares of 22 mm side) were set based on guidelines from the literature  [26, 38].

Three different sets of vibrotactile stimuli were designed and associated with the virtual buttons, aimed at simulating different materials and effects. The first two sets were designed starting from the vibration stimuli used in the classification experiment (Sect. 3.2), whereas the last set was designed based on the rendering capabilities of the device (Sect. 3.3).

Case study 1

Fig. 10

Spectra of the signals designed based on the original vibration stimuli from the classification experiment (dashed orange lines). For each material, two design techniques are shown: signals band-pass filtered in the 100–600 Hz range, and signals synthesizing a few relevant components of the original spectra (marked by blue vertical lines)

Fig. 11

Spectra of the stimuli shown in Fig. 10 as actually reproduced by the device. The blue vertical lines represent the main frequency components of the original signals

Based on the reported positive results of tactile material classification (Sect. 2), a first implementation of virtual buttons tested the straightforward reproduction of the same vibration stimuli used in the experiment. Unfortunately, such attempt was not effective at all: the original stimuli shown in Fig. 10 (orange lines) gave rise to weak and distorted reproductions, as visible in Fig. 11 (orange lines). Indeed, the chosen actuator can efficiently reproduce only a few concurrent spectral components, whereas the reproduction of rich spectral and dynamic content is generally unsatisfactory.

In an attempt to overcome such issue, two new sets of stimuli were prepared: the former consisted in a filtered version of the original signals, made using a tenth-order Butterworth filter with pass-band 100–600 Hz; the second set was synthesized by tuning the frequency and decay time of exponentially decaying sine oscillators to the most prominent components of the original signals in the same frequency band, that is two components at 115 and 470 Hz for wood, one component at 430 Hz for plastic, and two components at 230 and 550 Hz for metal. The RMS power of all stimuli was normalised within a 500 ms window, so as to make them uniform and maximize vibration amplitude while avoiding distortion. The signals from both sets are made available via an open-access repository.² Figure 10 shows the spectra of the obtained stimuli compared to those of the original recordings, while Fig. 11 reports the spectra as actually rendered by the device. Although the newly designed stimuli were in general better rendered than the original recordings, their reproduced characteristics are worth noticing: an artifact was introduced at around 100 Hz in all signals; lower frequency energy (< 100 Hz) was boosted in the synthesized plastic and especially metal stimuli, while both filtered and synthesized metal stimuli also gained energy at their fundamental frequency, being it close to the resonant frequency of the actuator; conversely, the first component of the original wood signal (115 Hz) was not reproduced by the device; finally, the spectra of reproduced wood and plastic were quite similar, as they have frequency components that are close to each other (430 Hz vs. 470 Hz).

User evaluation

The two sets of stimuli underwent each a separate subjective evaluation.

Three virtual buttons labeled A, B, and C were respectively linked to wood, plastic and metal stimuli, either filtered or synthesized, which were triggered by finger pressure exceeding 1 N. Given that the target use of the device is in generally noisy professional environments, and that the evaluation was performed in a silent room, an auditory distractor reproducing the noise of a crowded room (70 dB(A)) was continuously delivered during the assessment.

Fourteen subjects (6 male, 8 female) aged between 22 and 54 (M = 33.1; SD = 7.4) participated. Each participant performed two sessions, respectively evaluating three buttons using either filtered or synthesized stimuli. The task was to freely operate the buttons and answer an online questionnaire in Italian containing 7-point Likert scale evaluations and multiple choice questions:

• 1. Degree of difference among the three buttons, based on touch only. The evaluation scale ranged from ‘barely different’ to ‘very different’.

• 2. General tactile quality of all buttons. The evaluation scale ranged from ‘not appreciated’ to ‘much appreciated’.

• 3. Compliance of each button. Despite the fact that no displacement was rendered, compliance illusion could be elicited thanks to the vibrotactile response to finger pressing  [20, 33]. The evaluation scale ranged from ‘weak’ to ‘strong’.

• 4. Material each button was made of, among five options (metal, plastic, wood, glass, and rubber). There was one question per material, each with possible multiple choice of buttons and an additional ‘none’ option (i.e., A, B, C, none).

For the sake of clarity, in what follows the buttons reproducing filtered stimuli are referred to as Plastic Filtered (PF), Wood Filtered (WF) and Metal Filtered (MF), while those reproducing synthesized stimuli are labeled as Plastic Synthesized (PS), Wood Synthesized (WS) and Metal Synthesized (MS).

Fig. 12

Score distributions of perceived difference among the buttons in case study 1 (1 = barely different, 7 = very different)

Results

Figure 12 shows the perceived difference scores among the buttons. In addition, participants reported that buttons reproducing wood (WS, WF) and plastic (PS, PF) rendered similar stimuli, whereas buttons with metal feedback (MS, MF) differed from the others in both sets.

Fig. 13

Score distributions of tactile feedback appraisal in case study 1 (1 = not appreciated, 7 = much appreciated)

Concerning the appraisal of tactile feedback, the distributions reported in Fig. 13 show that the evaluations were more consistent for filtered rather than synthesized stimuli. However, nobody assigned the highest score to either filtered or synthesized stimuli.

Fig. 14

Score distributions of perceived compliance for each button in case study 1 (1 = weak, 7 = strong)

Regarding the perceived compliance, Fig. 14 reports for both sets high scores for stimuli related to metal (MS, MF) and low scores for stimuli related to wood (WS, WF). In general, the perceived compliance seemed to depend more on the simulated material than the type of stimuli (filtered or synthesized).

Fig. 15

Attribution of materials with filtered stimuli in case study 1

Fig. 16

Attribution of materials with synthesized stimuli in case study 1

Material attributions are reported for the two sets separately in Figs. 15 and 16, revealing high uncertainty in both cases. Notably, wood was the only material not attributed to any button by almost all participants: wood stimuli were mostly identified as plastic or glass, confirming our observations regarding the similar spectral content of the original wood and plastic signals. In general also material attribution seemed to be rather independent of the set type. Given the limited differences among the stimuli in terms of spectral content and components decay, this suggests that participants generally confirmed the same material attributions in both sets.

Case study 2

Fig. 17

Vibration waveforms of the four virtual buttons of case study 2, as measured by an accelerometer (see Fig. 9). Different sequences of sine waves are produced at finger-press onset and release, whose frequencies are reported in green and yellow bars respectively. Onset/release triggering forces are shown at the bottom

Fig. 18

Spectrograms of the vibrotactile feedback associated to the four virtual buttons

In the light of the poor overall results obtained in case study 1 with filtered and synthesized stimuli based on the original vibration recordings, a further set of signals was designed from the ground up making direct use of the piezo driver. Its internal synthesis engine can generate temporal sequences of sine waves at frequencies multiple of a fundamental of the piezo (about 7.8 Hz), thus limiting the design space. Four virtual buttons labeled A, B, C, and D were designed, aimed at simulating different tactile materials and effects. Based on known illusory kinesthetic effects elicited by vibrotactile feedback  [20, 33], some mechanical features of real buttons were also simulated. The main characteristics of the designed buttons are listed below:

• Button A simulates a silicon rubber key.³

  Onset: when the applied force exceeds 3 N, a sequence of two sine waves (5 cycles at 78 Hz and 3 cycles at 164 Hz) is synthesized producing a peak acceleration of $1.35\,\hbox {ms}^{-2}$. The frequency of the first signal is below the pass-band of the device, resulting in a “rubbery” tactile effect just before a further transient that simulates a soft ‘click’.

  Release: the same two waves are played in reverse order when the force drops below 1.2 N, producing a $1.9\,\hbox {ms}^{-2}$ peak acceleration.

  Together, these sequences simulate the acceleration curves resulting from pressing a finger on soft materials  [23].

• Button B simulates the behavior of a metal membrane switch.

  Onset: a strong transient consisting of a single cycle of a sine wave at 304 Hz with $3.8\,\hbox {ms}^{-2}$ peak acceleration is triggered when the applied force exceeds 1.6 N, simulating the sudden deflection of a metal membrane.

  Release: the same feedback is generated when the force falls below 1.2 N, resulting in $3.4\,\hbox {ms}^{-2}$ peak acceleration.

• Button C simulates a latching push button made of plastic, inspired by the switches found on old table lamps.

  Onset: when a 0.8 N force is exceeded, a 78 Hz sine wave is played for 150 ms, simulating the initial phase of button depression. Right after that, a stronger transient (a short 172 Hz sine wave) is produced with $1.8\,\hbox {ms}^{-2}$ peak acceleration, simulating a ‘click’.

  Release: when the applied force falls below 0.6 N, a short 164 Hz sine wave is generated to simulate the release ‘click’, resulting in $2.2\,\hbox {ms}^{-2}$ peak acceleration.

• Button D simulates a more abstract metal resonance with long decay, especially suited to long-press actions.

  Onset: when a 2.4 N force is exceeded, a strong 172 Hz sine wave with long decay is produced to simulate a ‘click’, and if pressure is held for more than 700 ms a further short feedback (250 Hz sine wave) is generated. The peak acceleration produced is $3.4\,\hbox {ms}^{-2}$.

  No feedback is provided on release.

Despite the fact that wood-related feedback scored best in the reported material classification experiment, no button was designed to simulate wood. This mainly because the strong low frequency components typical of this material can not be correctly rendered by the device, and secondly because it was not reputed a common material for buttons. Metal and plastic were instead found more appropriate, however since they represent the two materials that were more often confused in the classification experiment, one button rendering metal (D) was strongly differentiated by implementing longer decaying resonances.

As demonstrated in case study 1, the main spectral components of the vibratory signals used in the material classification experiment (see Fig. 3) cannot be accurately rendered on our device, given its limited bandwidth (see Fig. 8). The frequencies of the synthesized sine waves, as well as their decays and amplitudes, were therefore empirically chosen based on the pass-band of the device and informal testing, while leaving the generation of higher frequency components to the inherent harmonic distortion taking place with strong signals (see buttons B and D in Fig. 18). As a result, the main spectral components of the designed stimuli are generally at lower frequency than those in the stimuli used for the material classification experiment (Sect. 2). However such pitch-shift is known to have no effect on the perception of a specific material, being it more associated to the varying size of an object  [13]. Instead, materials were mainly defined by the designed decays (e.g., shorter for rubber and plastic), amplitudes (e.g., stronger for metal) and harmonic content.

The vibrotactile feedback produced by the buttons was measured by attaching a Wilcoxon 736 accelerometer on top of the touchscreen, between the virtual buttons (see Fig. 9). Figures 17 and 18 respectively show the waveforms and spectrograms of the feedback signals. The measured signals, as well as video footage of the four virtual buttons being operated are made available via an open-access repository.⁴

When vibrations were produced, the system emitted also some parasitic sound, however this was hardly perceivable in the (noisy) environment chosen for the device evaluation, and could therefore be ignored.

User evaluation

Sixteen subjects (9 male, 7 female) aged between 25 and 47 (${M}=34.7$; ${{SD}}=8.1$) evaluated the virtual buttons. The assessment took place in a realistic situation (i.e., a crowded open-space office hosting about 40 people), thus no additional auditory distractor was required. The task was to freely operate the buttons and answer an online questionnaire containing the same 7-point Likert scale evaluations and multiple choice questions proposed in the case study 1 (see Sect. 3.2.1).

Results

Fig. 19

Score distributions of the perceived difference among the four buttons in case study 2 (1 = barely different, 7 = very different)

Fig. 20

Score distributions of tactile feedback appraisal in case study 2 (1 = not appreciated, 7 = much appreciated)

As highlighted in Fig. 19, participants generally rated the buttons as clearly distinguishable from each other, furthermore they expressed general appreciation for the quality of tactile feedback, as shown by the score distributions in Fig. 20.

Fig. 21

Score distributions of the perceived compliance for each button in case study 2 (1 = weak, 7 = strong)

Evaluation ratings of the perceived compliance are reported in Fig. 21: The effect was most pronounced for button D followed by button B, while ratings related to buttons A and C are distributed in the lower and the mid-upper part of the scale.

With regard to the association of five materials (metal, plastic, wood, glass and rubber) with the virtual buttons, their choice distribution is reported in Fig. 22. Attributions mostly agreed with the intended feedback design (see Sect. 3.3): button A was mainly associated with rubber, button C clearly with plastic, and buttons B and D even more distinctly with metal. Concerning wood and glass—both not simulated—the former was not associated with any button by half of the participants, while a small group associated the latter almost uniformly with all the given possibilities, including the ‘none’ option.

General discussion

The two reported case studies revealed the challenges posed by the tactile rendering of well distinguishable virtual buttons on touchscreens.

Fig. 22

Attribution of materials in case study 2

The most relevant outcome of the two studies concerns the discrimination of buttons: although differences were generally perceived in both assessments, score distributions show that a careful design of tactile signals to exploit device’s peculiarities (e.g., resonances and damped frequencies, controlled distortion), in conjunction with the optimization of force thresholds at which feedback is provided, can be even more effective than the use of real vibration recordings, even if adapted to the device’s pass-band. Indeed, only in case study 2 buttons are rated as “very different” by most participants, suggesting that the reproduction of signals with realistic frequency components and decays is not sufficient to enable a precise discrimination. Discrimination in case study 2 may also have improved by rendering illusory cues related to button mechanics (e.g., switches, material compliance).

The tactile feedback generated by our device was generally appreciated in both studies, however the virtual buttons implementation of case study 2 received higher scores. After comparing the synthesized signals with the same signals provided via the analog input of the piezo driver, we can claim that the advantage of the internal synthesizer is all in its reduced design space which imposes to concatenate sine waves at frequencies that maximize the actuator’s efficiency.

A major difference between the two case studies concerns the attribution of materials to the virtual buttons.

In case study 1, stimuli originated from metal vibrations (MF, MS) were almost evenly assigned among the available materials, whereas in case study 2 the buttons inspired to metal properties (B, D) were clearly identified. Therefore, the design of effective stimuli simulating metal seems to be linked with long decay times and the inharmonicity of their spectra: Indeed, although buttons B and D in case study 2 render spectral components that differ from those in the original recording of metal vibration, they generate longer decaying resonances and inharmonic content typical of metal  [12].

The buttons designed to render plastic materials—that is, PF and PS in case study 1 and button C in case study 2—were correctly assigned by 50% and 62% of the participants, respectively in case study 1 and 2. In both studies, plastic was more confused with glass than other materials.

In general, wood was the material more associated with the ‘none’ option, which is indeed correct for case study 2. Surprisingly, in case study 1 the buttons rendering wood-related stimuli (WF, WS) were mostly associated to every other materials except wood. The impaired reproduction of frequencies below 100 Hz clearly explains these associations.

As mentioned above, vibrotactile feedback can be used to simulate to some extent button mechanics, thus incrementing differences among virtual buttons or easing material identification, as we did in case study 2. However, the illusion of kinesthetic feedback can be effectively elicited only if tracking the applied force, and by careful design and control of the delay and duration of the stimuli  [28].

Our design partially confirms the findings of Sadia et al.  [33], who investigated forces and accelerations involved in various button press actions (e.g., latch, push and toggle) and emulated such mechanics by reproducing tactile stimuli by means of piezo actuators. In particular, for their latch button a waveform pattern was generated whose temporal evolution is close to button C in case study 2. On the other hand, they triggered stimuli when the applied force was much greater than ours. To improve our latch button it would be possible to trigger the two parts of the stimuli onset based on multiple subsequent force triggers (e.g., at 3 and 10 N). Moreover, based on the dataset provided by Alexander et al.  [1], who characterized the physical properties of more than 1500 push buttons, it would be possible to design further button mechanics.

Conclusion

An experiment was reported that tested the ability to classify materials from impulsive auditory or tactile feedback, respectively reproduced via high-quality headphones and a low-quality haptic display. Besides confirming previous findings on the performance of auditory-based classification, our results prove that the use of degraded tactile feedback enable equivalently good classification while keeping low mismatch rates.

These outcomes inspired the design of virtual buttons on an ad-hoc implemented prototype device, consisting of a touchscreen interface which offers rich tactile feedback and force sensing. Virtual buttons were rendered by means of waveforms triggered at different pressing-force thresholds in the range of soft-touch. Two user panels evaluated several aspects of the tactile feedback associated to various sets of virtual buttons through a questionnaire. Although the buttons originated from the vibrotactile stimuli used in the classification experiment were relatively clearly discriminated (case study 1), participants could more successfully do so with buttons designed from the ground up exploiting the prototype device’s response and characteristics (case study 2). Moreover in case study 2 the association of buttons with materials was in good agreement with the original design intentions.

A further experiment is planned which will address the design and discrimination of (illusory) mechanical properties of buttons (e.g., switches) rendered through vibrotactile feedback only.

Based on our finding that vibrotactile cues enable a robust discrimination of different materials and other tactile properties, we suggest that the proposed design of virtual buttons may result in more effective operation of touchscreen interfaces in environments where auditory or visual distractors, as well as vibration noise generated by machinery, are present.