Professional environments often present acoustic and tactile disturbances as well as visual occlusion, and therefore machines with virtual buttons providing well differentiated tactile cues may support a more effective non-visual human-machine interaction
[11].
Prototype device
Supported by the experimental results on material categorization based on tactile cues, a prototype device was designed which generates vibrotactile feedback in response to touch interactions. The device implements virtual buttons triggered by variable pressing forces in a soft-touch range (0–5 N). The main goal of the device is to render virtual buttons that are easy to discriminate based on tactile cues only. As active touch enhances the sensitivity to vibrations
[29, 32], even better discrimination performance was expected compared to the reported experiment, which was conducted in passive conditions.
Figure 7 shows the layout of the prototype, built using off-the-shelf components.
The device displays virtual buttons on a 2.8 inches TFT touchscreen (see Fig. 9) whose capacitive layer locates finger contact positions, while the exerted pressing force is measured using a BND-611N load-cell (0–1 kg) placed at the bottom of the structure. The load-cell is driven by a 24 bit HX711 AD converter with a sampling rate of 80 Hz. Although techniques exist for the estimation of finger force during tapping actions
[33], a more direct and accurate measure via a low-cost load-cell was preferred. Moreover, the use of a force sensor allows tracking the release phase of pressing gestures before a finger loses contact with the touchscreen surface, which would not be reliable based only on capacitive or resistive sensing.
Vibrotactile feedback is generated by a Samsung Electro Mechanics (SEMCO) PHAH353832 piezoelectric actuator (dimensions L \(35\times \hbox {W}\) \(3.8\times \hbox {H}\) 3.2 mm, weight 2.7 g) controlled by a Texas Instrument DRV2667 piezo driver, connected to an Arduino Mega 2560 microcontroller board via the I2C communication bus. Compared to other haptic technologies (e.g., LRA, ERM and voice coil), piezo actuators can render fast transients at different frequencies. Other benefits are their small size and low power consumption.
The piezo driver may operate in analog mode, by amplifying (up to \(200\,{V}_{{pp}}\)) an audio-level signal at its analog input, or use the internal digital-controlled synthesis engine to generate simple sequences of sine waves, whose parameters (frequency, amplitude, attack and decay time, and duration) can be defined.
Since the touchscreen is suspended on a foam layer, the device is slightly compliant to external pressure (\(\le 1\,{{mm}}\)).
The frequency response of the system was measured in the range 50–1000 Hz by attaching a Wilcoxon 736 accelerometer on top of the touchscreen (center position). As shown in Fig. 8, the device is mostly efficient around the resonance frequency of the piezo actuator (230 Hz), whereas it is substantially unable to reproduce frequencies below 100 Hz. Concerning the upper part of the tested range, artifacts are present above 700 Hz, also resulting in audible distortion.
The device displays up to four virtual buttons labeled A, B, C, D (see Fig. 9), matching the number of main functions commonly found on professional appliances (2 to 6). Their shape and size (squares of 22 mm side) were set based on guidelines from the literature
[26, 38].
Three different sets of vibrotactile stimuli were designed and associated with the virtual buttons, aimed at simulating different materials and effects. The first two sets were designed starting from the vibration stimuli used in the classification experiment (Sect. 3.2), whereas the last set was designed based on the rendering capabilities of the device (Sect. 3.3).
Case study 1
Based on the reported positive results of tactile material classification (Sect. 2), a first implementation of virtual buttons tested the straightforward reproduction of the same vibration stimuli used in the experiment. Unfortunately, such attempt was not effective at all: the original stimuli shown in Fig. 10 (orange lines) gave rise to weak and distorted reproductions, as visible in Fig. 11 (orange lines). Indeed, the chosen actuator can efficiently reproduce only a few concurrent spectral components, whereas the reproduction of rich spectral and dynamic content is generally unsatisfactory.
In an attempt to overcome such issue, two new sets of stimuli were prepared: the former consisted in a filtered version of the original signals, made using a tenth-order Butterworth filter with pass-band 100–600 Hz; the second set was synthesized by tuning the frequency and decay time of exponentially decaying sine oscillators to the most prominent components of the original signals in the same frequency band, that is two components at 115 and 470 Hz for wood, one component at 430 Hz for plastic, and two components at 230 and 550 Hz for metal. The RMS power of all stimuli was normalised within a 500 ms window, so as to make them uniform and maximize vibration amplitude while avoiding distortion. The signals from both sets are made available via an open-access repository.Footnote 2 Figure 10 shows the spectra of the obtained stimuli compared to those of the original recordings, while Fig. 11 reports the spectra as actually rendered by the device. Although the newly designed stimuli were in general better rendered than the original recordings, their reproduced characteristics are worth noticing: an artifact was introduced at around 100 Hz in all signals; lower frequency energy (< 100 Hz) was boosted in the synthesized plastic and especially metal stimuli, while both filtered and synthesized metal stimuli also gained energy at their fundamental frequency, being it close to the resonant frequency of the actuator; conversely, the first component of the original wood signal (115 Hz) was not reproduced by the device; finally, the spectra of reproduced wood and plastic were quite similar, as they have frequency components that are close to each other (430 Hz vs. 470 Hz).
User evaluation
The two sets of stimuli underwent each a separate subjective evaluation.
Three virtual buttons labeled A, B, and C were respectively linked to wood, plastic and metal stimuli, either filtered or synthesized, which were triggered by finger pressure exceeding 1 N. Given that the target use of the device is in generally noisy professional environments, and that the evaluation was performed in a silent room, an auditory distractor reproducing the noise of a crowded room (70 dB(A)) was continuously delivered during the assessment.
Fourteen subjects (6 male, 8 female) aged between 22 and 54 (M = 33.1; SD = 7.4) participated. Each participant performed two sessions, respectively evaluating three buttons using either filtered or synthesized stimuli. The task was to freely operate the buttons and answer an online questionnaire in Italian containing 7-point Likert scale evaluations and multiple choice questions:
-
1.
Degree of difference among the three buttons, based on touch only. The evaluation scale ranged from ‘barely different’ to ‘very different’.
-
2.
General tactile quality of all buttons. The evaluation scale ranged from ‘not appreciated’ to ‘much appreciated’.
-
3.
Compliance of each button. Despite the fact that no displacement was rendered, compliance illusion could be elicited thanks to the vibrotactile response to finger pressing
[20, 33]. The evaluation scale ranged from ‘weak’ to ‘strong’.
-
4.
Material each button was made of, among five options (metal, plastic, wood, glass, and rubber). There was one question per material, each with possible multiple choice of buttons and an additional ‘none’ option (i.e., A, B, C, none).
For the sake of clarity, in what follows the buttons reproducing filtered stimuli are referred to as Plastic Filtered (PF), Wood Filtered (WF) and Metal Filtered (MF), while those reproducing synthesized stimuli are labeled as Plastic Synthesized (PS), Wood Synthesized (WS) and Metal Synthesized (MS).
Results
Figure 12 shows the perceived difference scores among the buttons. In addition, participants reported that buttons reproducing wood (WS, WF) and plastic (PS, PF) rendered similar stimuli, whereas buttons with metal feedback (MS, MF) differed from the others in both sets.
Concerning the appraisal of tactile feedback, the distributions reported in Fig. 13 show that the evaluations were more consistent for filtered rather than synthesized stimuli. However, nobody assigned the highest score to either filtered or synthesized stimuli.
Regarding the perceived compliance, Fig. 14 reports for both sets high scores for stimuli related to metal (MS, MF) and low scores for stimuli related to wood (WS, WF). In general, the perceived compliance seemed to depend more on the simulated material than the type of stimuli (filtered or synthesized).
Material attributions are reported for the two sets separately in Figs. 15 and 16, revealing high uncertainty in both cases. Notably, wood was the only material not attributed to any button by almost all participants: wood stimuli were mostly identified as plastic or glass, confirming our observations regarding the similar spectral content of the original wood and plastic signals. In general also material attribution seemed to be rather independent of the set type. Given the limited differences among the stimuli in terms of spectral content and components decay, this suggests that participants generally confirmed the same material attributions in both sets.
Case study 2
In the light of the poor overall results obtained in case study 1 with filtered and synthesized stimuli based on the original vibration recordings, a further set of signals was designed from the ground up making direct use of the piezo driver. Its internal synthesis engine can generate temporal sequences of sine waves at frequencies multiple of a fundamental of the piezo (about 7.8 Hz), thus limiting the design space. Four virtual buttons labeled A, B, C, and D were designed, aimed at simulating different tactile materials and effects. Based on known illusory kinesthetic effects elicited by vibrotactile feedback
[20, 33], some mechanical features of real buttons were also simulated. The main characteristics of the designed buttons are listed below:
-
Button A simulates a silicon rubber key.Footnote 3
Onset: when the applied force exceeds 3 N, a sequence of two sine waves (5 cycles at 78 Hz and 3 cycles at 164 Hz) is synthesized producing a peak acceleration of \(1.35\,\hbox {ms}^{-2}\). The frequency of the first signal is below the pass-band of the device, resulting in a “rubbery” tactile effect just before a further transient that simulates a soft ‘click’.
Release: the same two waves are played in reverse order when the force drops below 1.2 N, producing a \(1.9\,\hbox {ms}^{-2}\) peak acceleration.
Together, these sequences simulate the acceleration curves resulting from pressing a finger on soft materials
[23].
-
Button B simulates the behavior of a metal membrane switch.
Onset: a strong transient consisting of a single cycle of a sine wave at 304 Hz with \(3.8\,\hbox {ms}^{-2}\) peak acceleration is triggered when the applied force exceeds 1.6 N, simulating the sudden deflection of a metal membrane.
Release: the same feedback is generated when the force falls below 1.2 N, resulting in \(3.4\,\hbox {ms}^{-2}\) peak acceleration.
-
Button C simulates a latching push button made of plastic, inspired by the switches found on old table lamps.
Onset: when a 0.8 N force is exceeded, a 78 Hz sine wave is played for 150 ms, simulating the initial phase of button depression. Right after that, a stronger transient (a short 172 Hz sine wave) is produced with \(1.8\,\hbox {ms}^{-2}\) peak acceleration, simulating a ‘click’.
Release: when the applied force falls below 0.6 N, a short 164 Hz sine wave is generated to simulate the release ‘click’, resulting in \(2.2\,\hbox {ms}^{-2}\) peak acceleration.
-
Button D simulates a more abstract metal resonance with long decay, especially suited to long-press actions.
Onset: when a 2.4 N force is exceeded, a strong 172 Hz sine wave with long decay is produced to simulate a ‘click’, and if pressure is held for more than 700 ms a further short feedback (250 Hz sine wave) is generated. The peak acceleration produced is \(3.4\,\hbox {ms}^{-2}\).
No feedback is provided on release.
Despite the fact that wood-related feedback scored best in the reported material classification experiment, no button was designed to simulate wood. This mainly because the strong low frequency components typical of this material can not be correctly rendered by the device, and secondly because it was not reputed a common material for buttons. Metal and plastic were instead found more appropriate, however since they represent the two materials that were more often confused in the classification experiment, one button rendering metal (D) was strongly differentiated by implementing longer decaying resonances.
As demonstrated in case study 1, the main spectral components of the vibratory signals used in the material classification experiment (see Fig. 3) cannot be accurately rendered on our device, given its limited bandwidth (see Fig. 8). The frequencies of the synthesized sine waves, as well as their decays and amplitudes, were therefore empirically chosen based on the pass-band of the device and informal testing, while leaving the generation of higher frequency components to the inherent harmonic distortion taking place with strong signals (see buttons B and D in Fig. 18). As a result, the main spectral components of the designed stimuli are generally at lower frequency than those in the stimuli used for the material classification experiment (Sect. 2). However such pitch-shift is known to have no effect on the perception of a specific material, being it more associated to the varying size of an object
[13]. Instead, materials were mainly defined by the designed decays (e.g., shorter for rubber and plastic), amplitudes (e.g., stronger for metal) and harmonic content.
The vibrotactile feedback produced by the buttons was measured by attaching a Wilcoxon 736 accelerometer on top of the touchscreen, between the virtual buttons (see Fig. 9). Figures 17 and 18 respectively show the waveforms and spectrograms of the feedback signals. The measured signals, as well as video footage of the four virtual buttons being operated are made available via an open-access repository.Footnote 4
When vibrations were produced, the system emitted also some parasitic sound, however this was hardly perceivable in the (noisy) environment chosen for the device evaluation, and could therefore be ignored.
User evaluation
Sixteen subjects (9 male, 7 female) aged between 25 and 47 (\({M}=34.7\); \({{SD}}=8.1\)) evaluated the virtual buttons. The assessment took place in a realistic situation (i.e., a crowded open-space office hosting about 40 people), thus no additional auditory distractor was required. The task was to freely operate the buttons and answer an online questionnaire containing the same 7-point Likert scale evaluations and multiple choice questions proposed in the case study 1 (see Sect. 3.2.1).
Results
As highlighted in Fig. 19, participants generally rated the buttons as clearly distinguishable from each other, furthermore they expressed general appreciation for the quality of tactile feedback, as shown by the score distributions in Fig. 20.
Evaluation ratings of the perceived compliance are reported in Fig. 21: The effect was most pronounced for button D followed by button B, while ratings related to buttons A and C are distributed in the lower and the mid-upper part of the scale.
With regard to the association of five materials (metal, plastic, wood, glass and rubber) with the virtual buttons, their choice distribution is reported in Fig. 22. Attributions mostly agreed with the intended feedback design (see Sect. 3.3): button A was mainly associated with rubber, button C clearly with plastic, and buttons B and D even more distinctly with metal. Concerning wood and glass—both not simulated—the former was not associated with any button by half of the participants, while a small group associated the latter almost uniformly with all the given possibilities, including the ‘none’ option.
General discussion
The two reported case studies revealed the challenges posed by the tactile rendering of well distinguishable virtual buttons on touchscreens.
The most relevant outcome of the two studies concerns the discrimination of buttons: although differences were generally perceived in both assessments, score distributions show that a careful design of tactile signals to exploit device’s peculiarities (e.g., resonances and damped frequencies, controlled distortion), in conjunction with the optimization of force thresholds at which feedback is provided, can be even more effective than the use of real vibration recordings, even if adapted to the device’s pass-band. Indeed, only in case study 2 buttons are rated as “very different” by most participants, suggesting that the reproduction of signals with realistic frequency components and decays is not sufficient to enable a precise discrimination. Discrimination in case study 2 may also have improved by rendering illusory cues related to button mechanics (e.g., switches, material compliance).
The tactile feedback generated by our device was generally appreciated in both studies, however the virtual buttons implementation of case study 2 received higher scores. After comparing the synthesized signals with the same signals provided via the analog input of the piezo driver, we can claim that the advantage of the internal synthesizer is all in its reduced design space which imposes to concatenate sine waves at frequencies that maximize the actuator’s efficiency.
A major difference between the two case studies concerns the attribution of materials to the virtual buttons.
In case study 1, stimuli originated from metal vibrations (MF, MS) were almost evenly assigned among the available materials, whereas in case study 2 the buttons inspired to metal properties (B, D) were clearly identified. Therefore, the design of effective stimuli simulating metal seems to be linked with long decay times and the inharmonicity of their spectra: Indeed, although buttons B and D in case study 2 render spectral components that differ from those in the original recording of metal vibration, they generate longer decaying resonances and inharmonic content typical of metal
[12].
The buttons designed to render plastic materials—that is, PF and PS in case study 1 and button C in case study 2—were correctly assigned by 50% and 62% of the participants, respectively in case study 1 and 2. In both studies, plastic was more confused with glass than other materials.
In general, wood was the material more associated with the ‘none’ option, which is indeed correct for case study 2. Surprisingly, in case study 1 the buttons rendering wood-related stimuli (WF, WS) were mostly associated to every other materials except wood. The impaired reproduction of frequencies below 100 Hz clearly explains these associations.
As mentioned above, vibrotactile feedback can be used to simulate to some extent button mechanics, thus incrementing differences among virtual buttons or easing material identification, as we did in case study 2. However, the illusion of kinesthetic feedback can be effectively elicited only if tracking the applied force, and by careful design and control of the delay and duration of the stimuli
[28].
Our design partially confirms the findings of Sadia et al.
[33], who investigated forces and accelerations involved in various button press actions (e.g., latch, push and toggle) and emulated such mechanics by reproducing tactile stimuli by means of piezo actuators. In particular, for their latch button a waveform pattern was generated whose temporal evolution is close to button C in case study 2. On the other hand, they triggered stimuli when the applied force was much greater than ours. To improve our latch button it would be possible to trigger the two parts of the stimuli onset based on multiple subsequent force triggers (e.g., at 3 and 10 N). Moreover, based on the dataset provided by Alexander et al.
[1], who characterized the physical properties of more than 1500 push buttons, it would be possible to design further button mechanics.