Keywords

1 Introduction

Mid-air haptic (MAH) feedback aims at stimulating the sense of touch in mid-air and is predominantly rendered using ultrasound  [3]. Compared to vibrotactile feedback, MAH feedback is rendered with less fidelity, but its advantage of not requiring an actuator attached to the body makes it an attractive way of providing system feedback in gesture-based interfaces. While MAH’s usefulness for in-car infotainment  [9] or digital kiosks  [13] has already been studied, an important question remains: How can information be encoded within MAH patterns?

One way is to rely on MAH shapes, with studies showing shape identification rates ranging between 44%  [18] to 60%  [12] and 80%  [14]. However, the shape or pattern type constitutes only one channel of information. As MAH sensations are often applied to gesture-based devices as a type of system feedback, the possibility to encode information in different channels of the MAH sensation would be valuable as it enables richer and more diverse feedback messages. For instance, in a car, one MAH channel could convey the music loudness while another channel could convey the AC fan speed. To enable such applications, we investigated two potentials information channels regarding MAH sensations, namely intensity and velocity.

To this end, we estimated people’s abilities to discriminate between differences in the intensity and velocity of MAH patterns. We assessed the Just-Noticeable Difference (JND) of both the intensity and velocity of MAH sensations. In order to compare JNDs across stimulus types (MAH vs vibrotactile) and dimensions (intensity vs velocity), Weber fractions were calculated  [21]. Earlier work has observed Weber fractions for the intensity of vibrotactile stimuli to range between 13% and 16%  [5, 8]. When considering the velocity of tactile brushing stimuli, Weber fractions between 20% and 25% have been found   [7]. However, it is unknown how these values would extend to the case of MAH stimuli.

Additionally, we investigated to what extent age would influence the discrimination performance. We expected a decrease in performance with increasing age based on earlier work, showing a sharp drop in identification accuracy of MAH shapes with increasing age  [18]. Finally, we assessed participants’ confidence in their own discrimination performance, in order to explore whether their subjective experience would align well with their actual discrimination performance.

2 Methods

2.1 Participants

A total of 50 people, 25 men and 25 women, participated in the study, with a mean age of 44.58 years (SD = 15.93, range = 19–77). Only 7 participants were left-handed, all others were right-handed. Exclusion criteria were: Previous experience with mid-air haptic feedback and touch deficits in the upper limbs. This study was approved by the local social and societal ethics committee (G- 2018 10 1361), and participants received a voucher of 20 euro as incentive.

2.2 Device and Stimuli

The MAH stimuli were created using the sensation editor of the touch development kit, developed by Ultrahaptics©. The device produced a dial-like MAH pattern: one focal point, with amplitude modulated at 125 HzFootnote 1, which moved around a circular path with a diameter of 5 cm (see Fig. 1-right). In the intensity discrimination task, the target stimulus levels were intensities of 100%, 95%, 90%, 85%, 80%, 75%Footnote 2, and all stimuli had a speed of 2 rev/s. In the velocity discrimination task, the target stimulus levels were velocities of 2 rev/s, 1.8 rev/s, 1.6 rev/s, 1.4 rev/s, 1.2 rev/s, and 1 rev/s, and all stimuli had an intensity of 100%. In both task, the reference stimuli were characterized by an intensity level of 100% and a speed of 2 rev/s.

Fig. 1.
figure 1

Left: Experimental set-up. Right: Visual representation of the MAH sensation

Full size image

2.3 Tasks

Both experimental tasks were identical except for the target stimuli being used (see Device and Stimuli). The tasks were created using PsychoPy 3  [17] and were administered on a laptop. We applied the Method of Constant Stimuli in a 2-interval forced choice (IFC) task. In each trial, participants indicated which of two stimuli they perceived as the target stimulus, with the stimuli being presented successively in randomized order. In the intensity task, participants were instructed to indicate which stimulus had the lowest intensity, and in the velocity task participants had to indicate which stimulus had the slowest speed. Thus, the target stimulus was always the stimulus with the lowest intensity or slowest speed. Participants had to indicate the temporal position of the target stimulus by pressing “1” or “2” on their keyboard, with pressing 1 referring to the first stimulus they experienced in that trial, and pressing 2 to the second stimulus. No corrective feedback was provided. In line with the typical amount of stimulus levels involved  [19], we chose six different target stimulus levels to compare against the reference level (see Device and Stimuli). Each level was presented 12 times, resulting in a total of 72 trials in each task. Each stimulus had a duration of one second, with an inter-stimulus interval of 0.5 s, and unlimited response time. After each response, the trial ended when participants indicated on a scale from 1–7 how confident they felt about their answer.

2.4 Procedure

After signing the informed consent, participants filled out a demographic survey. They were asked to position their non-dominant hand above the MAH device, with their arm resting on the armrest to avoid excessive fatigue (see Fig. 1-left). The armrest ensured a stable distance of about 20 cm between the hand and the MAH device. Participants were urged to keep their hand horizontally and fixed above the device. To familiarize with the feeling of MAHs, they were presented with a single focal point on the palm during 10 s before starting with the tasks. Next, participants were told that all stimuli in both tasks would have the same pattern. To give them an idea of what to expect, a visual representation (short movie) of the dynamical stimulus pattern was presented on their screen during 5 s (see Fig. 1-right). Next, participants were assigned to both conditions in a counterbalanced order (within-participant design). Between the two discrimination tasks, a short break was provided.

3 Results

3.1 Discrimination of Intensities

Two participants were excluded from these analyses because of a technical error and a misunderstanding of the instructions, resulting in a sample of N = 48. We first checked whether the subjective confidence in one’s responses would correlate with one’s accuracy (=proportion of correct responses)Footnote 3, and this was indeed the case. A (repeated measures) correlation coefficient of \(r_{r_m}(239) = .63, p < .001, 95\%\ CI = [.54, .70]\) was observed (R package rmcorr  [1]). The correlation coefficient slightly increased in strength after removing nine influential data points based on Cook’s distance: \(r_{r_m}(230) = .66, p < .001, 95\%\ CI = [.58, .73]\). Secondly, we tested for an association between discrimination accuracy and age. A negative association between age and discrimination accuracy was observed, with \(r = -.35, t(46)= -2.53, p = .02, 95\%\ CI = [-.58, -.07]\). However, when we removed three data points that were identified as influential outliers, the correlation coefficient diminished in strength and could only barely reach statistical significance (see Fig. 2), \(r = -.30, t(43) = -2.03 , p = .049, 95\%\ CI = [-.54, -.002].\)

We modeled the count data (how many times the target stimulus was picked as having the lowest intensity) using a generalized linear mixed model (GLMM) with a probit link function, and with the intensity of the target stimulus as predictorFootnote 4 (R package lme4  [2]). The model included person-based random intercepts, as they significantly improved the model’s fit, showing a \(SD = 0.14\), \(X^2(1) = 16.43\), \(p < .0001\), with \(AIC = 1158\), \(BIC = 1169\), \(LL = -576\) for the model with random intercepts, and \(AIC = 1172.4\), \(BIC = 1179.8\), \(LL = -584.21\) for the model without random intercepts. The final model did not include random slopes, as this resulted in singular fit, which often indicates overfit. No influential outliers were observed, using R package influence.ME  [16].

Next, the JND was estimated from the GLMM model using the Delta method, without any lapse rate or bias parameters included in the model (R package MixedPsy  [15]). The JND was defined as the inverse function of the slope (see also Dallmann, Ernst, & Moscatelli, 2015  [6]). It measured the perceptual noise and was computed as half of the difference between the stimulus values at response probabilities of 75% and 25%. We obtained a JND = \(12.12\% (SE = 0.62\%)\), \(95\% CI = [10.90\%, 13.33\%]\). Next, a preliminary Weber fraction was calculated as follows: \(\Delta I/I = c\), \(12.12/100 = 0.1212\), resulting in a fraction of 12.12%  [11, 21].

Fig. 2.
figure 2

Scatter plots with regression line (grey zone indicates the 95% CI) showing the relationship between accuracy and age in the discrimination task involving intensities (left) and velocities (right).

Full size image

3.2 Discrimination of Velocities

No participants were excluded from this task, resulting in a sample of N = 50. A strong correlation was found between self-reported confidence in one’s ratings and accuracy, \(r_{r_m}(249)= .66, p < .001, 95\%\ CI = [.58,.72].\) The correlation coefficient increased in strength after removing 14 influential data points: \(r_{r_m}(235)= .72, p < .001, 95\%\ CI = [.65,.78].\) Similar to intensity discrimination, a negative relationship was observed between accuracy and age, \(r = -.31, t(48) = -2.29, p = .03, 95\%\ CI = [-.54, -.04]\), but disappeared after removing an outlier (based on Cook’s distance), \(r = -.23, t(47) = -1.65, p = .11 , 95\%\ CI = [-.48, .05]\) (see Fig. 2).

Next, we again modeled the count data (how many times the target stimulus was picked as having the slowest speed) using a GLMM with a probit link function, and with the target’s number of rev/sec as predictor. Using model comparison, we tested whether we should include random intercepts and slopes in the model. There was evidence for significantly improved model fit when adding person-based random intercepts, \(SD = 0.35, X^2(1) = 125.60, p < .0001,\) with \(AIC = 1175.1, BIC = 1186.2, LL = -584.55\) for the model with random intercepts, and \(AIC = 1298.7\), \(BIC = 1306.1\), \(LL = -647.35\) for the model without random intercepts. Given that the addition of random slopes resulted in a correlation of −1 between the random effects (often a sign of overfit), the final model only included random intercepts. No outliers were detected based on Cook’s distance. When estimating the discrimination threshold between different velocities based on the GLMM model and using the Delta method  [15] (no lapse rate or bias parameters were included in the model), we obtained the following values: JND = 0.51 rev/s (\(SD = 0.03\) rev/s), \(95\%\ CI\) = [0.45 rev/s, 0.56 rev/s]. A preliminary Weber fraction of 25.5% was obtained: \(\Delta V/V = c, 0.51/2 = 0.255 \).

4 Discussion

This study shows JNDs of 12.12% for the intensity and 0.51 rev/s for the velocity of MAH sensations. In other words, to use MAH intensity and velocity as information channels, a minimum difference in intensity of at least 12.12%, and a minimum velocity difference of at least 0.51 rev/s needs to occur from the reference output of 100% intensity and 2 rev/s.

The estimated Weber fraction of 25.5% for MAH velocity is comparable, although slightly higher, to the fractions observed for tactile brushing stimuli, which were found to vary between 20% and 25%  [7].

Our estimated Weber fraction for MAH intensity is based on acoustic pressure, while the Weber fractions for vibrotactile intensity are based on skin displacement, and therefore cannot be directly compared. Assuming the skin is mainly elastic  [22], displacement is proportional to stress. Additionally, stress at the MAH pattern is proportional to the square of the acoustic pressure  [4]. Thus, displacement is proportional to the square of the acoustic pressure and one can estimate the Weber fraction of MAH displacement to be 25.7% : \(\Delta I^2/I^2 = 0.257\). This estimated Weber fraction is higher than the Weber fractions observed for vibrotactile stimuli (ranging between 13%  [8] and 16%  [5]). We hypothesise that this difference lies in the nature of the stimuli. Vibrotactile stimuli are easier to discriminate when subtle changes in intensity or velocity occur, as they are rendered with higher fidelity.

Wilson et al.  [23] reported good motion perception of MAHs in the range 0.1–2 rev/s. Using our estimated Weber fraction, one could decrease the reference MAH velocity by 10 levels and still remain in this range. Similarly, previous studies reported a perceptual threshold for MAH intensities around 30% of the maximum output pressure  [10, 20]. Using our estimated Weber fraction, one could decrease the reference intensity by 9 levels and still remain above this threshold. Based on these observations, we can assume that one can discriminate between 11 and 10 levels of MAH velocities and intensities, respectively (i.e. 3.5 and 3.3 bits of information, respectively). Moreover, bits of information from two dimensions can be summed up  [11]. Hence we predict a maximum of 6.8 bits of information using the MAH pattern investigated in our study. However, we highlight that this value is only an upper bound for two reasons. Firstly, we relied on perceptual thresholds from the literature to estimate the bits of information, instead of data from an information transfer study. Secondly, these perceptual thresholds from the literature were observed for different types of MAH patterns, compared to the pattern we used in our study Therefore, while this estimate is encouraging for using MAH intensity and velocity as information channels, further research is needed to determine the actual value.

Interestingly, age was not found to be strongly related to discrimination accuracy of intensities or velocities. These findings are remarkable as earlier work found a strongly negative correlation between age and identification accuracy of MAH shapes, \(r = -.62\)  [18]. However, as we did not assess physiological characteristics of participants’ skin, it is hard to explain why the correlation between age and discrimination accuracy was weak in our study. Finally, based on the confidence ratings, it became clear that people’s subjective perception of their discrimination abilities aligned nicely with their actual performanceFootnote 5.

With this study, we performed a first step in mapping intensity and velocity JNDs of MAH patterns. However, the current study only looked at dial-like patterns. Future work should verify the generalizability of the obtained results regarding other pattern types. Moreover, we estimated the intensity and velocity Weber fractions based on only one reference stimulus, future work should improve this estimation by including multiple reference stimuli and target stimuli with values both lower/slower and greater/faster than the reference stimulus. Additionally, future studies should investigate to what extent the start and end positions of the stimuli in the velocity discrimination task could influence participants’ performance. To avoid a potential confound of start position, one could randomize the start position between trials. Despite these limitations, we believe the current study is a valuable initial step towards investigating intensity and velocity differences of MAH sensations as potential information channels.

5 Conclusion

This study is the first to report intensity and velocity JNDs for MAH sensations, which is essential when implementing MAHs as a means of system feedback. Based on our results, we recommend to only implement intensity differences greater than 12.12% and velocity differences greater than 0.51 rev/s. Intensity and velocity appear to be attractive information channels when considering MAH sensations as a means of system feedback. Future research could further elaborate on these findings, with a focus on investigating their boundary conditions.