Perceptually Compressive Communication of Interactive Telehaptic Signal

Abstract

During telehaptic applications over a shared communication medium, Weber’s law of perception based adaptive sampling scheme can be applied to reduce the data rate without degrading the perceptual quality of the haptic signal. However, the perceptual threshold (JND) is often unknown for a user. An experimental design involving bidirectional communication of haptic data over the Internet between the operator and teleoperator is carried out in this paper, which provides a real-time estimate of the Weber threshold for the telehaptic backward channel force signal. Using the proposed data-driven experimental protocol, we are able to reduce the packet rate significantly. We provide a subjective evaluation of the proposed technique to substantiate its usefulness.

1 Introduction

In recent years, the transmission of haptic data has received the attention of many researchers working in the area of telehaptics. This bidirectional transmission creates a global control loop between the operator and the teleoperator over a communication channel. Contrary to video or audio signal, in order to make the global control loop stable and to maintain the Quality of Service (QoS), a maximum communication delay of 30 ms is allowed [8]. Since the haptic signal is typically sampled at 1 kHz, to limit the packet rate while maintaining the QoS, perceptually significant adaptive sampling schemes based on Weber’s law of perception [9] or event detection [12] have been applied. Weber’s law of perception depicts a logarithmic relation between perceptual stimuli and human perception. Weber threshold is given by $\rho = \frac{\varDelta I}{I}$ where $\rho$ represents Weber threshold, $\varDelta I$ represents just noticeable difference (JND), i.e., the minimum change in signal magnitude required to produce a noticeable variation in signal perception and I is the reference signal magnitude.

In a perceptually significant adaptive sampling scheme, sampling of the kinesthetic force signal will be carried out at instances at which the relative difference exceeds the Weber threshold, i.e., $\frac{ \left| F_i - F_{i-1}\right| }{\left| F_{i-1} \right| }\ge \rho$ where $F_{i}$ represents force sample at time $t_i$ and $F_{i-1}$ represents the last perceived force sample at time $t_{(i-1)}$. Then these samples are transmitted. This helps in the reduction of packet rate. Hence the study of estimation of Weber threshold is essential. However, the JND depends on perceptual capabilities of an individual as well as the task one is performing.

A lot of research work has been carried out on the estimation of JND and Weber threshold. Though [5] discusses the estimation of JND on off-line data and their application in adaptive sampling, there is little research on the estimation of JND from real-time data. For the multidimensional signal, [5, 13] analyzed the variation in Weber threshold due to the change in force direction. Researchers have applied techniques like psychometric function [6, 15] mostly on off-line data to determine the Weber threshold of force signal. The work in [9] demonstrated adaptive sampling for telehaptic signal but with a fixed set of Weber threshold values. In this paper, we try to adaptively determine the Weber threshold directly from on-line data acquired in the given environment in real-time and simultaneously use it for adaptive sampling of the data on the backward channel. Haptics over Internet Protocol (HoIP) has been proposed in [7] to carry the haptic signal over the internet. [8] uses HoIP for the propagation of the telehapatic signal and also performs Weber sampling of the velocity signal. The stability of the global control loop is achieved through appropriate approaches, as specified in [10, 11, 14]. In this work, we assume that our global control loop satisfies the stability criteria, and we did not observe any instability of the global control loop during experimentation.

In this paper, we apply Weber’s law of perception based adaptive sampling scheme to the interactive telehaptic force signal under an appropriately simulated environment. The force signal is generated through signed distance field based haptic rendering of different watertight mesh models. This rendering is carried out at the teleoperator end, and the force signal is transmitted from the teleoperator to the operator on the backward channel. The force signal is adaptively sampled based on a real-time data-driven estimate of JND to reduce the packet rate over the Internet. In order to adaptively sample the haptic signal, we need to know the JND for the given task. We use an adaptively updatable probing threshold for sampling the signal on the backward channel while the user provides an intermittent task-nondisruptive feedback to the teleoperator through the forward channel. Our experiment showed that there is a substantial reduction in the haptic packet rate in-spite of not having any prior knowledge of the Weber threshold for the user. It is also well known that the user experience of haptic communication is dependent on network delay. We perform experiments to study this and show how it affects the users.

2 Proposed Method

2.1 Hardware Setup

The experiment is designed in two modes - 1) standalone and 2) networking. In standalone mode, only one haptic device is used with a computer. This mode is used only to study the effect of packetization delay and to serve as the reference (baseline experiment) during user studies. The theme of this paper is to carry out the experimentation in networking mode. Here two phantom omni haptic devices are used, one as an operator and the other as a teleoperator. These haptic devices are connected to two separate computers. These computers acting as end systems are connected through another node computer. A virtual network of these three computers is managed through Wide Area Network Emulator (WaNem) [3]. The unidirectional communication link capacity is set at 1000 Mbps (to avoid any network congestion), and the maximum packet rate is set to 1000 packets per second. The operator device is kept at a distance of 40 cm from the shoulder of the user. Arm and wrist are kept at fixed positions while perceiving the force. To analyze the effect of propagation delay $P_d$ on the estimated Weber threshold, one-way $P_d$ is emulated on the system and is varied between 0–15 ms, increasing in steps of 5 ms. The average packetization delay is observed to be 0.13 ms in the operator side and 0.45 ms in the teleoperator side, which is much lower than the introduced propagation delay $P_{d}$.

2.2 Signed Distance Field Based Haptic Rendering

To generate the force signal at the teleoperator, haptic rendering of watertight objects like bunny [1] and teddy [2] are carried out. The bunny model used in the experiment has 4976 faces and 2490 vertices, and it is voxelized into 135 $\times$ 142 $\times$ 90 grid cells, whereas the teddy model has 3192 faces and 1598 vertices. The teddy model is voxelized into 117 $\times$ 143 $\times$ 97 voxels. Given a haptic interaction point $H_{IP}$ and the object, the shortest distance between the point and the bounding object surface is obtained. Depending upon the sign of the distance $\phi (H_{IP})$, we can determine whether the interaction point lies inside $(\phi (H_{IP}) > 0)$/outside $(\phi (H_{IP}) < 0)$/on $(\phi (H_{IP}) = 0)$ the surface of an object.

The signed distance field is created for an object as in [4] and stored on a three-dimensional grid. During rendering, the $H_{IP}$ position is first obtained, and depending upon the sign of distance $\phi (H_{IP})$, the presence of $H_{IP}$ inside/outside the object is detected. If $H_{IP}$ lies inside the object, force $F(H_{IP})$ is generated from the gradient using a suitable scaling constant k as follows, $F(H_{IP}) = k \phi (H_{IP}) \nabla \phi (H_{IP})\left\| \nabla \phi (H_{IP}) \right\| _{2}^{-1}$.

2.3 Proposed Sampling Scheme

In this section, we discuss the proposed framework for adaptive sampling of force signal along with simultaneous Weber threshold estimation for an individual user.

As shown in Fig. 1, at the teleoperator, haptic and visual renderings of a solid object are carried out to simulate the teleoperation. The operator holds the device stylus and remotely explores the object at the teleoperator’s side through the means of video and haptic information transmitted from teleoperator to operator. This interaction is captured and transmitted in the form of position and velocity on the forward channel to the teleoperator. The end-effector running at the teleoperator end generates a force signal which is then adaptively sampled, and if its relative magnitude is greater than the Weber threshold $\rho$ at that instant, it is transmitted back to the operator on the backward channel. However $\rho$ is not known apriori and needs to be estimated from the interaction itself.

Fig. 1.

Proposed scheme for real-time interaction driven adaptive sampling.

We start with a very small value of $\rho$ (let us call it $W_r$) at the beginning and use it to adaptively sample the data. Since the user experience at such a small value of $W_r$ will not impede the perception of the user, after Ts interval $W_r$ is incremented by 5% of its previous value, and the adaptive sampler is changed accordingly. This continues until the operator’s experience is impaired. At this point, the operator sends a feedback signal (setting the FLAG bit TRUE in Fig. 1) through the forward channel to the teleoperator end. At the teleoperator end, when the FLAG is found to be TRUE, it is understood (see comments later) that the impairment in perception is due to loss of perceptually important samples and hence the threshold value $W_r$ is decremented agressively by 20% of its current value while passing the signal through the adaptive Weber sampler before sending it through the backward channel after packetization. At the operator end, if the user experience is not compromised then the threshold $W_r$ is again raised slowly by 5% else it is further decremented by 20%. The process continues until the teleoperation ends.

A few comments are in order at this point. We implicitly assume that (a) there is no communication delay during teleoperation as it introduces a hysteresis behaviour between exerted and perceived forces, (b) the control loop under which the teleoperation is performed does not suffer from transparency related impairments and (c) any perceptual impairment during teleoperation is due to a higher choice of the Weber threshold $W_r$ when the loss in information received at the operator end becomes perceptually significant.

It is well accepted in the literature that the Weber threshold $\rho$ lies in between $0.04 \le \rho \le 0.20$. Hence instead of starting the teleoperator with a value of $W_r$ to be zero, we set $W_{r_{min}} = 0.04$. As the $W_r$ value is slowly incremented, we saturate $W_{r_{max}}$ at 0.20, if required. The choice of an incremental step size of 5% was adhoc, but was found to be a good compromise between being aggressive or sluggish in approaching $W_r \rightarrow \rho$ (true value of the threshold) when the user continues to enjoy the interaction perceptually. A higher choice of increment requires the user to provide feedback more frequently, thus distracting the user away from the assigned task. Similarly, the choice of the decremental step size to be 20% is motivated by the fact that the system should quickly come out of the impaired deadzone of sampling to reduce the scope of continuation of perceptual impairment on overall user experience. Hence the choice of step sizes is such that perceptual impairment happens quite infrequently for the benefit of the user.

Now we address the question on how does the user attend to the problem of perceptual impairment. To start with, assume that the user is always in contact with the object to be operated on, it is expected that there will be always some force feedback and there will be some variation in it due to user interaction. When $W_r$ is small, the user will feel such changes more often and when $W_r$ becomes very high (say, close to 0.20), the user will not be experiencing much changes while still manipulating the object, thus impairing the experience. An immediate reduction in the $W_r$ tries to make the changes in force feedback a lot more perceivable. In the event the end-effector is not in contact with the user, any choice of value for $W_r$ is good as there will be no force feedback and $W_r$ will rapidly approach the value $W_{r_{max}}$.

We now discuss how the FLAG is set to TRUE during the communication over the forward channel. The default value of FLAG is FALSE when $W_r$ is continuously incremented by 5% at the backward channel. We attach a sound detector (a microphone that detects any audio utterance by the user, up to an appropriate choice of threshold to take care of ambient noise) to the user. Whenever the user experience worsens, the user is required to make some utterances which when detected, sets the FLAG to TRUE momentarily before being reset to FALSE. Thus the variable FLAG works on a monostable mode. When the received packet header at the teleoperator end has a FLAG value TRUE, the threshold $W_r$ is decremented by 20%. It may be mentioned that we use HoIP (Haptics over Internet Protocol), as proposed in [7], which has the provision of padding such flags in the packet header for bilateral communication. It may also be noted that one may instead use a call button feedback or a vision based experience impairment detection technique through user specified gesticulation. However, we have found that the audio based technique is less distractive and has comparatively a lower latency without impairing the dexterity of the operator.

The Weber threshold at a given instant depends on the transmission of the FLAG sequence through the forward channel, which in turn, depends on the operator feedback. As is common in any telehaptic communication, we use a sampling rate of 1 kHz, generating up to a maximum of 1000 packets per second when no Weber threshold based adaptive sampler is used. A user can provide her feedback as per her experience on interaction while the Weber threshold $W_r$ is updated every Ts. We select $T = 2$ s or $T = 3$ s, and within this interval, the value of $W_r$ is held constant.

Fig. 2.

Computed $W_{r}(t)$ and haptic packet rate for a given user for the object (top) bunny and (bottom) teddy with $T = 2$ s and $P_d = 0$ ms.

3 Experimental Results

Data Collection: The data is collected voluntarily from 16 male users following the standard ethical clearance of the university and the process of acquiring the data through informed consent of the users. The users, all right handed, were in the age-group of 22–34 years, out of which 10 users were unaware of the working of any haptic device before starting the experiment. The users did not have any history of neurophysiological illness. At the beginning of the experiment, every user was introduced to the device, and an appropriate training was given to get them familiar with the device to perceive the kinesthetic force signals properly. The experiment was also explained to them in detail.

The experimental protocol was as follows. 3D point cloud data of some solid objects are rendered for haptic interaction using a standard distance field based method. The user is requested to move the interaction stylus in the YZ plane only (Y being the vertical direction), so that the rendered force is along the Z (away from the body) direction only. This allows us to avoid user sensitivity to directional change in force, if any [5]. The rendered force is maintained within the range [0, 3N]. At the emulator level, the propagation delay $P_{d}$ is set at one of the following four values - 0, 5, 10 and 15 ms (see Sect. 2.1). While sending the force on the backward channel, the Weber threshold based sampling was done. The user orally gives the feedback whenever his interaction is perceived to be poor, as explained earlier. The Weber threshold $W_{r}$ is updated at 2 s and 3 s interval (T) in two different experiments.

The total time required for the experiment is approximately 1 h per user per rendered object. To avoid physical or device fatigue, users were asked to perform the experiment in multiple sittings with each sitting not exceeding 20 min. During experimentation, users were asked to listen to music using headphone to avoid distractions in the laboratory, if any. Users were also asked to evaluate the experiment on a 5 point scale, with 5 being excellent and 1 being very poor, for different values of $P_d$ and T based on the perceptual parameters: smoothness and experience [8]. The users were asked to rate the experiment relative to the baseline experiment (Sect. 2.1) on the standalone system with zero propagation and packetization delay and with no Weber sampling.

Fig. 3.

(top) Computed $W_{r}(t)$ and haptic packet rate for the object teddy with $T = 3$ s and $P_d = 0$ ms. (bottom) Effect of varying propagation delay $P_d$ on estimated Weber threshold $W_{r}(t)$ for the object teddy with $T = 3$ s for the same subject.

Results: For a given user, the computed $W_{r}$ as a function of time is plotted for two different objects (bunny and teddy) in Fig. 2 for a propagation delay $P_{d} = 0$ ms and backward channel update time $T = 2$ s. We observe that in both cases, the estimated Weber threshold $\rho$ is quite reasonable (about 7%) and follows a similarly repetitive pattern. The saving in packet rate is quite substantial (about 20 packets per second are being transmitted) to warrant the use of the proposed method. The corresponding SNR of the received force signal is above 30 dB, possibly good enough for most haptic interactions.

Table 1. Mean opinion score (MoS) and standard deviation (SD) on user study.

As we slow down the backward channel update time T to 3 s, we notice from Fig. 3 (top) that the nature of the corresponding plot of $W_{r}(t)$ remains more or less the same. However, the updating being slow, as expected, there is a marginal increase in packet rate. Ideally the values of Weber threshold ($\rho$) will be slightly less than the $W_{r}(t)$ value at which the user provides a negative feedback. In Fig. 3 (bottom), we plot only the local peaks of $W_{r}(t)$ function against feedback index (at which the user responded). We repeat the experiment for the same user and for the same object for different values of $P_{d}$ to study the effect of propagation delay. It may be observed that even though the haptic interaction becomes poorer as $P_{d}$ increases (see Table 1), the computed value of $\rho$ remains fairly unchanged, substantiating the appropriateness of the proposed method.

We present the result of subjective analysis of user experience for all 16 users for various values of propagation delay $P_{d}$ and update interval T on two specific perceptual parameters, namely interaction smoothness and user experience, in Table 1. It can be seen that the user experience gets quite impaired if the propagation delay increases to 10 ms and beyond, suggesting that the proposed technique should work under the upcoming 5G communication, but not on the current 4G technology. We also note that both the suggested update intervals $T = 2$ s or $T = 3$ s are acceptable to the user.

4 Conclusions

In this paper, we demonstrated an experimental protocol on how to reduce the interactive haptic signal packet rate over the Internet between the operator and teleoperator when the Weber threshold is not known. However, it is observed that as the propagation delay increases, the immersiveness of the virtual system decreases. However, within the specified QoS of 5 ms delay of 5G technology, the proposed method will work well. Hence the method has the potential for application in telehaptic application over a short-haul communication. However, the stability of the proposed telehaptic loop under the quantization error and delay jitter requires further investigation. One also needs to study the suitability of the proposed method for two handed interactions when the Weber threshold may be different for each hand.