Midair Tactile Reproduction of Real Objects

Abstract

Midair tactile display using ultrasound radiation pressure is suitable for tactile reproduction because of its reproducibility and controllability. This paper is the first report that compares the tactile feelings of real objects with those associated with artificial stimuli reproduced through a sequence of sensing, processing, and reproduction. We previously proposed the concept of a midair tactile reproduction system and examined the basic properties of the sensing part, but had not achieved the tactile display to the human skin. In this paper, we report a practical method for the pressure reproduction from the measured data and examine the accuracy of the reproduced stimulation. The psychophysical experiments evaluate the fidelity of the tactile reproduction for some objects such as brushes, sponges, and towels.

Supported in part by JSPS Kakenhi 16H06303 and JST CREST JPMJCR18A2.

1 Introduction

An airborne ultrasound tactile display (AUTD) [1, 2] can present tactile sensations with high spatio-temporal controllability in a non-contact manner. This advantage is suitable for tactile reproduction of real objects. However, the fidelity of tactile reproduction performance has not been studied in detail since the development of the device in 2008 [1]. In this study, we achieve a midair tactile reproduction system which consists of the AUTD and a tactile array sensor.

The concept behind the system was first introduced in our previous work-in-progress paper [3]. Further, we examined the design of the sensor and the reproduction scheme in [4]; however, the reproduced pressure distribution was strongly distorted owing to problems in the reproduction algorithm.

This paper is the first report that compares the tactile feelings of real objects to those associated with artificial stimuli reproduced through a sequence of sensing, processing, and reproduction. As a key part of the system, we report a practical method of the pressure reproduction. Conventional methods [4, 5] have no constraints on the upper limit of the output amplitude, and sometimes, it is impossible for devices to output the optimization results. In this study, we acquire a feasible solution by executing the Levenberg–Marquardt algorithm (LMA) [6, 7] with the constraint that the amplitudes of the transducers must all be equal. After confirming the basic physical performance of pressure reproduction, the fidelity of the reproduced sensation was evaluated in comparison with that of real objects by psychophysical experiments.

2 Tactile Presentation by Ultrasound Phased Array

2.1 Acoustic Radiation Pressure

Midair tactile stimulation using ultrasound is based on the acoustic radiation pressure. This pressure is a non-linear acoustic phenomenon which generates DC positive pressure on the boundary between the two types of media with different acoustic impedances. The acoustic radiation pressure P for plane ultrasound wave is described as $P=\alpha \frac{p_{0}^{2}}{\rho c^{2}}(>0)$, where $\rho$ is the density of the medium on the incident side; c is the speed of sound in the medium; $p_0$ is the RMS sound pressure of the ultrasound; and $\alpha$ is a constant that is dependent on the power reflection coefficient R such that $\alpha \equiv 1+R$, and $1\le \alpha \le 2$. In this study, we consider acoustic radiation pressure on solid surfaces in standard air; hence, $\rho =1.25\; \mathrm {kg/m^3}, \;c=340\;\mathrm {m/s}$, and $\alpha =2$.

2.2 Sound Field Generated by Phased Array

In this study, a desired pressure distribution is reproduced by the radiation pressure. We control the radiation pressure field by controlling the sound field.

Assuming a steady sinusoidal wave, a complex sound amplitude vector $\textit{\textbf{p}}$ generated by a phased array is expressed as follows [5]:

$$\begin{aligned}&\textit{\textbf{p}}=\textit{\textbf{G}}\textit{\textbf{q}}, \end{aligned}$$

(1)

$$\begin{aligned}&\textit{\textbf{p}}={[p_1, p_2,\dots ,p_M]}^{\top },\; p_m=p(\textit{\textbf{r}}_m)=p_{\mathrm {amp},m}e^{j\psi _m}, \end{aligned}$$

(2)

$$\begin{aligned}&G_{mn}=C\frac{D(\theta _{mn})}{|\textit{\textbf{r}}_{m}-\textit{\textbf{r}}_{n}|} e^{-\beta |\textit{\textbf{r}}_{m}-\textit{\textbf{r}}_{n}|} e^{jk|\textit{\textbf{r}}_{m}-\textit{\textbf{r}}_{n}|}, \end{aligned}$$

(3)

$$\begin{aligned}&\textit{\textbf{q}}={[q_1,q_2,\dots ,q_N]}^{\top },\;q_n=A_{n}e^{j\phi _n}. \end{aligned}$$

(4)

Here, $p_m(m=1,\ldots ,M)$ is a complex sound amplitude at the m-th control point $\textit{\textbf{r}}_m$, and $q_n(n=1,\ldots ,N)$ is a complex amplitude of the surface vibration velocity of the n-th transducer at $\textit{\textbf{r}}_n$. $\textit{\textbf{G}}$ is the transfer matrix, where C is a constant, $D(\theta )$ is the directivity function of a transducer, $\theta _{mn}$ is the angle between the transducer normal and $\textit{\textbf{r}}_{m}-\textit{\textbf{r}}_{n}$, $\beta$ is the attenuation coefficient and k is the wave number.

2.3 Controlling Pressure Field by Phased Array

In the case of tactile presentation, the driving signal of the phased array $\textit{\textbf{q}}$ is decided based on the sound pressure amplitude distribution $\textit{\textbf{p}}_{\mathrm {amp}}$ converted from the desired instantaneous pressure distribution $\textit{\textbf{P}}$. The proposed method solves the following problem; $\mathrm {find} \, \textit{\textbf{q}} \, \mathrm {s.t.} \, \textit{\textbf{p}}_{\mathrm {amp}}=|\textit{\textbf{Gq}}|$ at each time. The tactile presentation is performed by outputting the time series of $\textit{\textbf{q}}$ in a quasi-stationary manner. The phases of the complex sound distribution $\textit{\textbf{p}}$ are unconstrained. A more efficient output can essentially be achieved by modifying these phases [8].

3 Tactile Sensing

In order to measure a slight contact force of around 0.01 $\mathrm {N/{cm}^2}$($\simeq$the maximum presentation force per 1 AUTD) at sufficient sampling rate, we adopted the new tactile sensor that we had designed previously [4] using high-sensitivity microphones for use in this study. The sensor is composed of a $4\times 4$ channels microphone array (Fig. 1a). The contact pressure of the sensor surface causes pressure change in the sensor cavity and the microphone detects the change (Fig. 1b). Using this sensor, we can acquire a pressure distribution of $4\times 4$ channels at 11 mm intervals by a sampling frequency of 1 kHz. The frequency characteristics are compensated for components with $20-470$ Hz in this study. The components under 20 Hz and over 470 Hz are removed because of the low sensitivity of the microphones.

In order to achieve high fidelity of tactile sensation by this system, the vibrations produced on the tactile sensor must be similar to the actual vibrations on the human skin. Therefore, the mechanical characteristics of the sensor surface are required to be similar to those of the human skin. HITOHADA$\circledR$  gel sheet (hardness: Askar C0, thickness: 1 mm) of EXSEAL Co., Ltd., Japan is used as the contact surface. This is a super-soft urethane sheet comparable to human skin. Moreover, human fingerprints affect the generation and detection of slip. In this study, we compared the reproduction fidelity of tactile sensation between two conditions of the contact surface shape: non-treated condition and concentric circle groove pattern imitating fingerprints, as shown in Fig. 2.

4 Tactile Reproduction

4.1 Data Preprocessing

Before the optimization of the driving signal of AUTD, four preprocesses are performed on the measured data. First, half-wave rectification of the measured data is performed because the radiation pressure is always positive. Though this process generates unnecessary harmonics, we adopt it for the easiness and clearness of the process. Second, the pressure distribution data P is converted to sound pressure amplitude distribution $p_{\mathrm {amp}}$ by $p_{\mathrm {amp}} = \sqrt{2}p_{0} = c\sqrt{\frac{2\rho P}{\alpha }}$.

In order to limit the effects of pressures outside the region of interest, 20 control points are added around the region at 11-mm intervals ($4\times 4 \rightarrow 6\times 6$ points), and the sound pressure amplitudes are set to zero at the peripheral points. Subsequently, the 36-point data with 11 mm intervals are converted into 256-point data with 11/3 mm intervals by linear interpolation of the original data on the assumption that the pattern is smooth.

4.2 Optimization of Driving Signal of AUTD

In this study, the LMA [6, 7] was used to optimize the driving signal of AUTD. LMA is an iterative algorithm for solving non-linear least squares problems. In general, to solve a problem $\min _{\varvec{\theta }}\Vert \textit{\textbf{f}}(\varvec{\theta })\Vert _{2}^{2}$ by LMA, the parameter $\varvec{\theta }$ is updated to $\varvec{\theta }^{k+1} = \varvec{\theta }^{k} - [(\textit{\textbf{J}}^{k})^{\top }\textit{\textbf{J}}^{k} + \lambda ^{k}\textit{\textbf{I}}]^{-1}\textit{\textbf{f}}(\varvec{\theta }^{k})$, where $\lambda$ is the damping parameter and $\textit{\textbf{J}}$ is the Jacobian matrix of $\textit{\textbf{f}}(\varvec{\theta })$.

We solved the problem using the LMA with the constraint that the amplitudes of the transducers $A_1,\ldots ,A_N$ must all be equal. In this case, the optimization problem is described as follows:

$$\begin{aligned} \min _{\varvec{\theta }} \; \left\| \begin{array}{c} \mathrm {Re}(G\textit{\textbf{q}}-\mathrm {diag}(\textit{\textbf{p}}_\mathrm {amp})\textit{\textbf{u}}) \\ \mathrm {Im}(G\textit{\textbf{q}}-\mathrm {diag}(\textit{\textbf{p}}_\mathrm {amp})\textit{\textbf{u}}) \end{array} \right\| _{2}^{2} \end{aligned}$$

(5)

$$\begin{aligned} \textit{\textbf{q}} = A\times {[e^{j\theta _1}, e^{j\theta _2}, ..., e^{j\theta _N}]}^{\top },\textit{\textbf{u}} = {[e^{j\theta _{N+1}}, e^{j\theta _{N+2}}, ..., e^{j\theta _{N+M}}]}^{\top } \end{aligned}$$

(6)

where Re and Im are the real and imaginary parts of the complex vector, respectively. The results of the LMA depend on the initial value of $\varvec{\theta }_0$ because the algorithm determines only a local minimum. In this study, we use a zero vector or a final value of a previous time frame as the initial value.

Fig. 1.

Structure of (a) all the $4\times 4$ elements and (b) the single element of the sensor [4].

Fig. 2.

The concentric circle pattern of the sensor surface. (a) The whole pattern. (b) The enlarged view. (c) The appearance of the sensor.

Fig. 3.

An example of the simulated optimization results: The white circle indicates the position of each channel. They were optimized with 996 transducers (4 AUTDs), 256 control points at 50 iterations.

5 Experiments

5.1 Numerical Simulation

First, the reproduction performance of arbitrary static pressure distribution was evaluated via numerical simulation. We generated binary distributions on the control points such that the sound pressure amplitude of each channel was 0 or 1. The number of distributions was $2^{16}-1$ in total for 16 control points, with the exception of a distribution in which all the amplitudes were zero. Subsequently, we reproduced these distributions based on the point source model using MATLAB software. The examples of the simulated optimization results are shown in Fig. 3. The average error $\frac{1}{M}\sqrt{\Vert \varvec{Gq-p}\Vert _{2}^{2}/\Vert \textit{\textbf{p}}\Vert _{2}^{2}}$ was approximately $5-15$%.

Fig. 4.

The results of the reproduction of AUTD stimuli recorded by the sensor. Top: Original data. Bottom: Reproduction data. In (a), the top-left corner: 0 ch, bottom-right corner: 15 ch. In (b), Black solid line: 0 ch, black dotted line: 15 ch, and gray line: other channels.

5.2 Reproduction of Dynamic Radiation Pressure Distribution

Due to half-wave rectification and optimization errors in the reproduction algorithm, the waveform may change to some extent in the actual reproduction. Therefore, in the preliminary experiment, the dynamic pressure distribution generated by AUTD was reproduced by the system and again recorded by the sensor. Thus, the degree of distortion was examined and the distortion was corrected.

The presented stimuli were i) 1-point amplitude modulation (AM) stimulus to the 0 channel position of the sensor, ii) 2-point AM stimulus to the 0 and 15 channel positions, and iii) 1-point AM containing multiple frequency components to the 0 channel position. The AM frequency of i) and ii) was 100 Hz. As for iii), the AM 25 Hz waveform and the AM 250 Hz waveforms were added together, where the amplitude ratio was 3:2. Except for the first-time frame, 20 iterations of the LMA were performed using a final value of a previous time frame as an initial value each time. The other settings of equipment layout and optimization were the same as those of the numerical simulation.

As a result, the system reproduced some features of the original distributions, such as the peak position, the waveform and the AM frequency (Fig. 4a, 4b). Though there were some distortions caused by half-wave rectification, especially in iii), we left these distortions as they were. The amplification factor between the AUTD-driving voltage and the observed sensor output voltage is determined experimentally. We determined it so that the focused beam to the sensor just below the AUTD center produces an expected output amplitude.

5.3 Psychophysical Experiment

In the psychophysical experiment, the tactile sensations of real objects were reproduced and evaluated. This system can reproduce weak pressure of temporal frequency components over 20 Hz, so it is assumed that soft, light, and unsmooth objects are suitable. Therefore, the three types of objects, namely, soft brush, sponge and towel were selected. First, the waveform generated when tracing the sensor by each object was recorded for five times using an automatic stage as shown in Fig. 5a. The speed is approximately 5 cm/s and the contact surface is of circular shape, approximately 2 cm in diameter. This measurement was performed in each condition of the sensor surface with and without concentric circle grooving treatment. The reproduction stimulation was then generated on a palm according to the proposed method. Simultaneously, amplitude correction was carried out based on the result of the preliminary experiment.

The experiment was carried out in two stages. First, the ability of the real object to discriminate tactile sensation was confirmed. The tactile sensation of the real object was randomly presented 12 times to the blindfolded participants by a human hand. They answered which object was presented each time. Next, the reproduction stimulation was evaluated. In practice, each type of reproduced stimulation was once presented without the prior information of what object was reproduced. Subsequently, the remaining three types of stimulation were randomly presented four times. The number of trials was minimized considering the fatigue of the participants. For each stimulus, participants were asked to answer two types of questions, i.e., A) how much they were similar to the tactile sensation of the brush/sponge/towel (1: totally different $\sim$ 7: very similar), and B) which object they felt it was closest to. This experiment was carried out twice by changing the surface treatment conditions. The participants were 10 people in their 20’s, two of whom were females, and all stimuli were presented on the palm of the dominant hand (one male was left-handed). While presenting the reproduction stimulus, the participants were listening to white noise by a headphone, and were allowed to touch the real objects freely.

Fig. 5.

(a) The measurement of the tactile sensations of the real objects. (b) The presentation of the reproduction stimulation.

Fig. 6.

The responses to the question “which object they felt it was closest to.” The number indicates the number of responses.(b)/(c) is the result of the groove-less/grooved sensor surface condition.

Fig. 7.

The F-measure calculated from the responses to the question B.

Fig. 8.

The similarity to the tactile sensation of real objects that the participants answered.

Fig. 9.

The power spectrum of the original tactile sensation data at the four major channels that the objects passed through (2, 6, 10, 14 ch). These are all after calibration and half-wave rectification.

As shown in Fig. 6a, the real objects could be discriminated with a correct answer rate of over 95%. In contrast, the correct answer rate of the reproduction stimulus was lower(Fig. 6b, 6c). In Fig. 7, the F-measure was calculated from the answer to question B) shown in Fig. 6b, 6c. As the result indicates, the F-measure exceeds 0.33, which is the chance rate. Moreover, a paired t-test did not show a significant difference between non-treated condition and concentric circle treated condition($t(2) =1.60, p=.25$). In addition, a three-way repeated measures ANOVA was carried out on the answers to question A). As shown in Fig. 8, there is also no significant difference in the subjective similarity evaluation by the surface treatment condition.

Based on these results, it was confirmed that three types of tactile sensation could be discriminated to some extent by this system without prior information. However, there were no significant differences of subjective evaluation and discrimination results between two surface conditions. Therefore, it is assumed that the reproduced stimulation contains some important tactile features, regardless of the surface treatment, for identification of the three types of objects. As shown in Fig. 9, the intensity increased by the surface groove, especially in the components under 300 Hz. In both surface conditions, the intensity of high-frequency band over 300 Hz was largest in the towel data, followed by the sponge data, and then the brush data. It was suggested that the intensity of the high-frequency band was the clue for identification of the three types of objects.

It must be noted that some important tactile features in the original stimulation might have been lost owing to the spatial resolution of 11 mm of the tactile sensor. The other possible issues include the acoustic streaming, lack of tangential force, and shortness of the displayed spatial resolution owing to the limitation of the ultrasound wavelength.

6 Conclusion

In this study, we proposed an accurate tactile reproduction system by introducing the LMA for determining the driving signals of transducers on the AUTD. We evaluated the fidelity of the reproduced sensations of three typical elastic objects. As a result, the system could reproduce similar tactile feelings to a certain degree and display significant differences among them.

A crucial physical problem is the surface of the sensor that evaluates the tactile information of real objects. We evaluated the effect of the sensor surface characteristics. Regarding the tested objects, there was no significant difference between the two types of surfaces, i.e., the one with a concentric circle groove pattern and the other with no groove pattern.

The most demanding future work is to improve the spatial resolution of the sensor. The system can be used as a useful tool to clarify the human tactile sensation and as a practical system that records and reproduces the tactile feelings associated with various products.