Can Stiffness Sensations be Rendered in Virtual Reality Using Mid-air Ultrasound Haptic Technologies?

. Mid-air haptics technologies convey haptic sensations without any direct contact between the user and the interface. A popular example of this technology is focused ultrasound. It works by modulating the phase of an array of ultrasound emitters so as to generate focused points of oscillating high pressure, which in turn elicit haptic sensations on the user’s skin. Whilst using focused ultrasound to convey haptic sensations is becoming increasingly popular in Virtual Reality (VR), few studies have been conducted into understanding how to render virtual object properties. In this paper, we evaluate the capability of focused ultrasound arrays to simulate varying stiﬀness sensations in VR. We carry out a user study enrolling 20 participants, showing that focused ultrasound haptics can well provide the sensation of interacting with objects of diﬀerent stiﬀnesses. Finally, we propose four representative VR use cases to show the potential of rendering stiﬀness sensations using this mid-air haptics.


Introduction
Focused airborne ultrasound is nowadays the most mature and popular technology able to provide mid-air haptics.Arrays of ultrasonic transducers can produce phase-shifted acoustic waves which constructively interfere at points in space called focal points and destructively interfere elsewhere, conveying haptic sensations by varying acoustic radiation pressure on the skin.Focused ultrasounds have been already employed in several applications of Virtual Reality (VR) [4,5,9,10].However, despite the recent popularity of mid-air ultrasound technologies, to the best of our knowledge, no study has analyzed if and to what extent ultrasound haptic arrays can provide effective stiffness sensations.Most work using this technology in VR has been in fact dedicated to the rendering of shapes [6,9] and textures [1].
This work studies the capability of mid-air ultrasound haptics of rendering stiffness sensations when interacting with virtual objects.More specifically, we aim at identifying the differential threshold for stiffness perception when using a focused ultrasound array to render objects in VR.Of course, it is important to highlight that we are not rendering force feedback as if the user was interacting with a real piston.Our objective is to understand whether we can elicit/simulate stiffness sensations using focused ultrasound arrays.Our paper comprises a perceptual evaluation as well as four VR use cases (see Fig. 3), where we show the potential of our approach as an alternative to contact haptic feedback [2,11] in VR scenarios.

Experimental Setup
To validate focused ultrasound as a tool to provide stiffness sensations in VR, we prepared an experimental setup enabling participants to interact with 1-D stiffnesses.The setup is shown in Fig. 1.The virtual environment was composed of a virtual piston placed on a black table.The real environment was composed of an Ultrahaptics STRATOS platform, which is a commercial focused ultrasound array.It comprises a 16×16 planar array of transducers emitting 40 kHz ultrasound in an upward direction.The virtual environment was shown to the participant through an HTC Vive VR headset.A HTC Vive Tracker was attached to the dominant wrist of the participants to track the motion of their hands, and a virtual hand avatar mimicked this motion in the virtual environment.Finally, an HTC Vive controller was held by the participant in their non-dominant hand to answer the in-screen perceptual questions, and a pair of noise-canceling headphones avoided potential effects due to auditory cues arising from the device operation.

Haptic Rendering
The task consisted in comparing the stiffness of two virtual pistons.Each virtual piston was modeled as a 1-D spring following Hooke's law.Whenever a user enters in contact with the piston, the system simulates a spring-like feedback, where the pressure commanded by the Ultrahaptics device is defined by p = k(z 0 − z) + p 0 if the user contacts the piston, 0 otherwise.k is the simulated stiffness of the piston (in Pa/m, sound pressure over displacement), z the current altitude of the piston, z 0 = 30 cm its resting position, ∆z = z 0 − z its current compression, and p 0 = 146.87dB SPL (441 Pa) the absolute detection threshold we registered at 30 cm (when ∆z = 0).The piston is fully compressed at z = 20 cm (∆z max = 10 cm).The Ultrahaptics device generates localized pressure at a designated focal point.We rendered this point at the centroid of the upper plate of the piston (see Fig. 1).When the user interacts with the piston, this point results at the center of the user's palm as well as the center of the ultrasound array.As soon as the user contacts the piston at its resting position (z = 30 cm, ∆z = 0 cm), the device starts generating a pressure on the palm.This pressure increases as the user presses the piston down, reaching its maximum when the piston is fully compressed (z = 20 cm, ∆z max = 10 cm).The STRATOS platform can provide a maximum of 163.35 dB SPL at z = 20 cm and p 0 thus represents 15% of the maximum power.We rendered the focal point with the Ultrahaptics device using spatiotemporal modulation (STM), introduced by Kappus and Long [8].In STM, focal points are generated with a fixed frequency (usually the maximum achievable by the device, i.e., 40kHz).Since this frequency is very high, it poses significantly fewer constraints on the temporal evolution of the peak intensity and focal point position.Frier et al. [3] have investigated the trade-off between pattern repetition rate in STM and perceived intensity.A study on human's detection of focal points and basic shapes rendered via focused ultrasound stimuli can be found in [4].

Experimental Procedure
Participants were first required to fill out a pre-experiment questionnaire.Then, they were asked to wear the HTC Vive headset, tracker, controller (see Fig. 1).Participants had to compare two pistons with different rendered stiffness, modeled by a 1D spring law, as detailed in Sec.2.2.At the beginning of each interaction, the virtual environment only showed a transparent hand, marking the target hand position for starting the task.Before the start of the first interaction, the virtual hand of the participant was calibrated to ensure it matched the transparent hand along the three different axes.Once participants placed their hand at the starting position, the piston appeared right below it.This calibration phase prevents from too many wrist motions of the user's hand since the motion between the starting position and the piston is straightforward along the vertical axis.Participants were then requested to touch the top of the piston and press it down.As soon as the hand contacted the upper plate of the piston, the latter became green.
Participants were asked to press onto the piston until it was fully compressed, which was indicated by the piston becoming red.At this point, participants moved the hand up, releasing the piston.After that, they were asked to interact in a similar way with a second piston.After this second interaction, participants were finally asked to judge which of the two pistons felt stiffer.One piston served as a reference, displaying a reference stiffness k ref , while the other piston displayed a variable stiffness k test .After preliminary testings, we considered 6 values of test stiffness k test to be compared with 3 values of reference stiffness k ref .
The three stiffness values of the reference piston were:

Conditions and Experimental Design
Two conditions are considered in our experimental design: -C1 is the difference of stiffness between the reference piston and the test piston, -C2 corresponds to a binary variable, which is true if the piston perceived as the stiffest is indeed the one rendered with a higher stiffness constant.
The order of presentation of the two pistons was counterbalanced to avoid any order effect: every couple of pistons was presented in all orders.The starting reference was also alternated to ensure that fatigue did not influence the last block.Thus, participants were presented with 90 trials per reference stiffness (270 in total), divided in 5 blocks of 6 trials in a randomized order for each block.The experiment lasted approximately 40 minutes, with breaks between the blocks.

Participants and Collected Data
Twenty participants (16 males, 4 females) took part to the experiment, all of whom were self-identified right-handed.18 of them had previous experience with haptic interfaces.All were naive with respect to the study objectives.The age range of the participants was between 21 and 29 years (M=24).
For each couple of pistons, we collected as an objective measure the participant's answer.This answer corresponds to the piston (first or second) which was reported by the participant as the stiffest.The measure was then collected as a true discovery rate if the answer corresponds to the stiffest value rendered.Participants also completed a subjective questionnaire.The first set of questions was asked three times, after the 5 blocks dedicated to one reference stiffness, using

Results
Reference stiffness: k ref,1 .Answers to the questionnaire regarding confidence (Q1) ranged from 2 (nearly very unconfident) to 7 (very confident) out of 7, with a mean of 4.75 and standard deviation (SD) 1.2.Regarding fatigue (Q2), answers ranged from 1 (not fatigued) to 6 (moderately fatigued) out of 7, with a mean of 4.1 (SD=1.3).When the user was asked how easy the task was (Q3), answers ranged from 4 (slightly easy) to 7 (very easy) out of 7, with a mean of 5.45 (SD=0.9).The reported realness of the piston (Q4) ranged from 1 (not real at all) to 6 (moderately real) out of 7, with a mean of 3.6 (SD=1.5). Figure 2a shows the psychometric curve as well as the mean and standard deviation for each comparison piston.We obtained a JND value of 20% using a 75% threshold, along with a Point of Subjective Equality (PSE) of 2.16%.Reference stiffness: k ref,2 .Answers to the questionnaire regarding confidence (Q1) ranged from 2 to 6 out of 7, with a mean of 4 and SD 1. (Q2), answers ranged from 1 to 6 out of 7, with a mean of 4.25 and SD 1.5 .When the user was asked how easy the task was (Q3), answers ranged from 3 to 7 out of 7, with a mean of 5.05 and SD 1.3.The reported realness of the piston (Q4) ranged from 1 to 6 out of 7, with a mean of 4 and SD 1.6.Figure 2b shows the psychometric curve as well as the mean and standard deviation for each comparison piston.Under this reference, a 75% differential threshold of 32% was obtained with a PSE of 3.65%.
Reference stiffness: k ref, 3 .Answers to the questionnaire regarding confidence (Q1) ranged from 2 to 7 out of 7, with a mean of 4.1 and SD 1.3.Regarding fatigue (Q2), answers ranged from 2 to 6 out of 7, with a mean of 4.3 and SD 1.4.When the user was asked how easy the task was (Q3), answers ranged from 2 to 7 out of 7, with a mean of 5.25 and SD 1.4.The reported realness of the piston (Q4) ranged from 2 to 6 out of 7, with a mean of 3.94 and SD 1.5.Figure 2c shows the psychometric curve as well as the mean and standard deviation for each comparison piston.Differently from the other curves, this time we were not able to reach proportions of correct answers close to 1 on the right-hand side of the curve.This result could be due to the fact that the considered reference stiffness k ref,3 requires pressures close to the device maximum, i.e., 161.41 dB SPL when ∆z = z max , which is the 80% of the device power range.For this reason, it was not possible to test large increments.Another explanation for this behavior could be the presence of refractions and artifacts generated by the acoustic waves, which become more intense as the peak pressure increases and interfere with the overall perception of stiffness.This latter point is supported by how users described the haptic sensation over the three reference conditions.In fact, while during experiments on reference stiffness k ref,1 and k ref,2 users most often reported to feel a "circular shape", during experiments on k ref,3 users started to report feeling "lines" or "bars".The focal point generated by the device should remain circular at all intensity levels.For this reason, we evaluated the psychometric curve only taking into account the stiffness intensities for which users reported feeling a circular shape (blue points in Fig. 2c).Under this reference, a 75% differential threshold of 18% was obtained with a PSE of 0.58%.Post-experiment Questionnaire.All users were able to detect that the force increased over the displacement.However, only 48% of the participants were able to feel that the minimum pressure they felt when they first interacted with the piston was always the same (146.86dB SPL, 15% of the full range).When asked if the sensation they felt reassembled stiffness (Q5), 80% of users said that it did.The remaining 20% could not express what they felt, but still recognized an increase in force.When asked to describe the sensations they felt over the duration of all 60 pistons, answers ranged from "feeling a real piston" to feeling a "stream", "circular air flows", and "some kind of resistance".

Use Cases in Virtual Reality
We demonstrate the viability of rendering stiffness sensations using ultrasound focused arrays through four use cases in Virtual Reality, shown in Fig. 3 4 .
The first use case (see Fig. 3a) represents a scene at a carnival fair.It is composed of a stand at a carnival fair, featuring a pump, a release button, and a balloon to be inflated.Users are asked to inflate a balloon by repeatedly pressing on the pump.Every time the pump is pressed, it becomes a little stiffer to render the increased pressure inside the balloon.The second use case (see Fig. 3b) is composed of a small piano placed on a table.Piano keys are generally weighted having a higher stiffness for the lower register and a lower stiffness for the higher register.We render four different octaves, each having variable degrees of stiffness.Users are able to select a different set of octaves by pressing a button next to the piano.The third use case (see Fig. 3c) is composed of a hospital room with a virtual patient lying upon a bed.A 2-cm-wide area on the patient's stomach was rendered stiffer than the rest.Users are instructed to palpate the users stomach and indicate where they feel the stiffer region.The fourth use case (see Fig. 3d) is composed of four blocks that need to be pressed in a certain sequence to open a door containing a treasure chest.Each block has a different stiffness.To access the treasure, users must press the blocks in order of stiffness, from the lowest to the highest.On top of the door, there are four lights, that indicate the progress of the task.

Discussion and Conclusions
Ultrasound haptics is considered a very promising technology, as it is able to convey compelling haptic sensations without any direct contact between the user and the interface.However, as only recently ultrasound arrays have become available, very few works have studied the type of haptic sensations we can render with this technology.This work evaluates whether it is possible to render stiffness sensations in Virtual Reality using haptic feedback generated by ultrasound focused arrays.To calculate the JND and the PSE for this type of stiffness sensation, we carried out a human subject study enrolling 20 participants.Subjects were asked to compare the perceived stiffness of multiple virtual pistons, whose stiffness was rendered by an Ultrahaptics device via ultrasound haptic stimuli.In the literature, researchers have shown that the JND for stiffness discrimination can range from 8 to 23% [7,12].Jones and Hunter [7] have reported an average JND of 23% for participants comparing the stiffness of springs simulated using two servo-controlled electromagnetic linear motors.Each motor was coupled to one wrist of the subject.Tan et al. [12] calculated the JND of stiffness for a task which required grasping two plates with the thumb and index fingers and squeezing them along a linear track.A force which resisted the squeeze, simulating different levels of stiffness, was generated by an electromechanical system.When subjects had to squeeze the plates always for a fixed displacement, the JND registered was of 8%; on the other hand, when the displacement was randomized from trial to trial, the JND was of 22%.Of course, all these works rendered stiffness by providing kinesthetic feedback.
In our study, we found JND of 17%, 31%, and 19% for the three reference stiffness values 7358 Pa/m, 13242 Pa/m, 19126 Pa/m (sound pressure over displacement), respectively.The subjective questionnaires show that most subjects indeed identified the provided haptic sensations as stiffness.These results prove that it is indeed possible to simulate stiffness sensations using ultrasound haptic feedback in VR.Four use cases showed the potential and viability of our approach in immersive VR applications.Despite these promising results, our study has some limitations.First, it is important to stress that the haptic sensations rendered by ultrasound arrays is of course different than the haptic sensations usually felt when pressing a piston.For this reason, our objective is to simulate stiffness sensations.Another drawback is that the behavior we registered when commanding pressures higher than 162.43 dB SPL (90% of the maximum power of the device).The circular focal point started to feel like something different (a "line", a "bar") and the stiffness recognition rate significantly degraded.This is an issue we plan to address in the future, studying what happens from an acoustics point of view and understanding what it means in terms of human perception.

Fig. 1 :
Fig. 1: (Left) Setup: subjects interact with the Ultrahaptics interface while wearing a HTC Vive display.The dominant hand is tracked using a Vive Tracker while the other hand holds a Vive Controller to answer the questions.The virtual scene with the piston is shown as an inset.(Right) Visual representation of the focal point and its relation with the displacement of the piston.
a 7-item Likert scale: (Q1): I felt confident when choosing the response after each interaction; (Q2): After the experiment, I felt tired; (Q3): The task was easy; (Q4): It felt like pressing a real piston.Then, at the end of the experiment, we asked them to answer two open questions: (Q5): Would you describe what you felt as stiffness?If not, please attempt to describe it; (Q6): Do you any have any further comment or suggestion?

1 PSE - 3 . 2 :
65% JND -31.65% Change of stiffness (% with respect to reference k ref,2 ) Proportion of correct answers (b) reference stiffness k ref,2 = 13424 Pa/m Change of stiffness (% with respect to reference k ref,3 ) Proportion of correct answers (c) reference stiffness k ref,3 = 19126 Pa/m Fig. Psychometric curves for the three reference stiffness values, fitting a cumulative Gaussian to the data.We plot the proportion of correct answers in function of the percentage increase in stiffness with respect to the reference one k ref,1 .The vertical dashed and solid lines represent the PSE and the 75% differential threshold.Error bars represent standard deviation.
(a) Carnival fair (b) Piano playing (c) Medical palpation (d) Dungeon quest Fig. 3: We implemented four use cases in Virtual Reality.We render different stiffness sensations using the ultrasound stimuli generated by our Ultrahaptics interface.