Keywords

1 Introduction

Tele-operation is a technology in which a remote robot is controlled from a distance by a human operator. This technology is helpful for performing tasks in environments that are dangerous (e.g. nuclear power plants), unreachable (e.g. space or deep sea), or require scaling (e.g. keyhole surgery) [8]. Many of these tasks require high levels of dexterity, and are executed in environments that cannot be fully predicted. Therefore, full automation is impossible, and thus the human needs to be in the loop.

In daily life, humans perform dexterous tasks such as picking up an object effortlessly and efficiently. A tight control is kept over the ratio between the load force perpendicular to the object’s surface and the grip force normal to the object’s surface, both during static holds [11], and during arm movements [2]. One of the sensory cues that helps to keep this safety margin small is slip force, as micro-slips are acted upon reflexively to restore a proper safety margin [5]. In tele-operation systems, slip force feedback is generally lacking, which could be a reason why dropping or crushing an object is much more likely in remote interactions than in direct interactions [3].

Several slip feedback displays have been developed (for instance [10]), and it is known that vibratory slip feedback improves task performance in virtual remote interactions [9]. Nonetheless, it is hard to integrate these systems in real-time tele-operation tasks with real remote environments. Most remote robots are only equipped with simple force sensors which are not able to measure slip. Pachierotti et al. do show that slip feedback improves performance in a real tele-operated setup [6], but they used grounded haptic devices to track only two finger tips. Many real tele-operation systems have complex input devices, such as gloves, which interfere with integrating large slip feedback devices. Since we did want to test real remote interactions with full hand tracking, we chose to focus on the effect of simple, abstract warning signals about slip on the user’s behavior. We measured the participant’s grip force profiles, and compared auditory and haptic vibration feedback to a condition with no slip force feedback. This allowed us to test if slip feedback affected the timing and the magnitude of the forces that participants used while holding a real object with a tele-operated arm.

2 Material and Methods

2.1 Participants

Fifteen healthy participants were recruited, of which two were authors (KG and PdV). All participants but the authors were naive to the purpose of the experiment. Four participants failed the initial stereo-vision test (\({}{}{}{}{}30^{\prime \prime }\) on the TNO test for stereoscopic vision, Laméris Instrumenten B.V), resulting in 7 men and 4 women, aged \(35\pm 10\) years (mean ± s.d.) completing the full experiment. All participants were right-handed, provided written informed consent, and were compensated for their time. Ethical approval for the experiment was provided by the TNO Institutional Review Board (#2021-036). One of the data sets was corrupted due to a technical failure, so 10 data sets were used for analysis.

2.2 Setup

A custom-designed tele-operation system was used in this experiment, as shown in Fig. 1. The remote robot consisted of a KUKA LBR iiwa robot arm (KUKA Aktiengesellschaft, Augsburg-Germany), to which a SHADOW robot hand (SHADOW ROBOT COMPANY LTD. London-United Kingdom) was attached. In this experiment, the Kuka arm was fixed in a convenient position, and only the robot hand was actuated. The standard SHADOW fingertips were replaced with BioTac sensors (SynTouch Inc, Montrose, CA-USA), which provide 3 DOF impedance measurements in Volts. These impedance values can be converted to forces by a per-sensor calibration, as shown in [7], but BioTac does not provide such a calibration. As we were not interested in absolute forces, but only in differences, this was not a problem for our experiment. Nonetheless, it should be noted that whenever ‘forces’ are presented in this paper, these actually are impedances. A TNO-developed tele-presence unit housed a stereo-camera on a movable platform which filmed the remote robot. The movements of the platform were synchronized with the motion of the TNO-developed Head-Mounted Display (HMD), to which the filmed data was streamed. To control the remote robot, participants wore a Senseglove exoskeleton glove (Senseglove B.V., Delft-the Netherlands). The Senseglove’s finger positions and orientations were mapped to the Shadowhand’s positions and orientations. The Senseglove was used to provide passive kinesthetic feedback in all conditions using magnetic brakes on its tendon system, and active feedback in the haptic condition using the integrated ERM vibration motors on the back of the finger tips (C1026B002F, Vybronics, Inc., Shenzen-China. Motors rated 150 Hz, 1.5 mm).

Fig. 1.
figure 1

Schematic overview of the setup, with equal colours indicating systems that were synchronized between participant on the left and remote robot on the right. The Kuka arm was set in a convenient fixed position, so only the Shadowhand reacted to the user’s motion (for clarity, only 2 of the 5 fingers of the Shadowhand are shown). A rope, attached to the bottom of the water bottle, was lead through a pulley underneath the bottle, and lead back to the experimenter for slip induction.

Full size image

2.3 Testing Auditory and Haptic Latencies

Because of the complexity of the full tele-operation system, we could not measure the communication latencies between all parts of the system. However, since a confounding factor for our specific research question could be a difference in system latency between the onsets of haptic and auditory feedback, we tested this prior to running our experiments. We used a stereo plug to record data from two contact microphones (AD-35 transducer, Otraki, Manila - Philippines) simultaneously. One microphone was taped to the Senseglove vibration motor, and the other to a speaker. We sent out two signals simultaneously, and measured the response at 41.1 kHz. From the recordings, spectrograms were calculated, as shown in Fig. 2. Auditory onsets were defined as the power spectrum 450 Hz exceeding \(-50\) dB. Haptic onsets were defined as the absolute raw signal exceeding 0.035. Averaging the differences between these onsets across 10 repetitions showed that the haptic signal onset lagged the auditory one by \(44\pm 26\) ms. To avoid this confound in the main experiment, we added a 44 ms delay in software between slip detection and onset of the auditory feedback.

Fig. 2.
figure 2

Recordings from stereo contact microphones to test onset latencies between haptic and auditory stimuli. Left column: raw recordings, right column: spectrograms. The vertical dashed lines indicate determined onset times. For this recording, the measured haptic signal lagged the auditory one by 30 ms. Note that the measurement is only meant to determine the difference in onset times, so t = 0 has no specific meaning.

Full size image

2.4 Paradigm

Before commencing the experiment, all participants were asked to take a stereo-acuity test. Upon successful completion (score \(< {}{}{}{}{}30^{\prime \prime }\)), participants were familiarized with the control of the tele-operation hand using the Senseglove by free exploration. When they felt comfortable with controlling the hand, they were asked to put on the HMD and continue exploration. Haptic and auditory feedback was demonstrated to avoid startle reactions, and whenever the participant felt ready to proceed, the main experiment was started.

In the main experiment, the participant’s task was to hold a remote object with the tele-operated hand. The object was a compressible, partially-filled water bottle (total weight 250 g). Aluminum plates were attached to its sides to provide a flat surface for sensor readings from the robotic hand, and a rope was attached to its bottom. Participants were asked to only use their index finger and thumb to hold the object, while keeping the other fingers curled towards the palm. At the start of each trial, the experimenter placed the object in the tele-operated hand, and the participant adjusted their grip to a firm hold, without excessively squeezing the object. Next, the experimenter introduced a vertical disturbance on the object by pulling on the rope. The participant was asked to keep a firm grasp on the object, without it slipping and without excessive squeezing. The slip data gathered from the robot’s sensorized fingers was used to provide slip feedback to the user. A derivative of the force tangential to the object’s surface was calculated, as a quick change in vertical force would indicate slip. Whenever slip velocity exceeded a pilot-determined threshold in tangential force velocity (i.e., a sensor-specific impedance velocity), the object was considered to be slipping and feedback was initiated. Three slip feedback conditions were tested: (1) auditory feedback using 450 Hz pure tone, (2) haptic feedback using Senseglove’s standard vibration signal at 50% of the maximum amplitude, and (3) no slip feedback. In all conditions, kinesthetic haptic feedback was provided by mapping normal forces detected on the remote robot’s fingers to a gentle resistive force on the Senseglove tendons. In addition, visual feedback of the object and part of the rope were also present in all conditions. Although the visual feedback of the rope tension provided some feedback about the timing of the disturbance, the magnitude of the force disturbance was varied by the experimenter, so the time between disturbance start and start of the slip was hard to predict for participants. Whenever the object was dropped before the participant could respond to the disturbance, the trial was rejected and repeated.

All trials of each slip feedback type were presented as a block, with ten repetitions of the same feedback type making up one block. When a block was completed, the experimenter verbally administered the 6-item NASA TLX questionnaire to assess the participant’s cognitive load on a 21-point scale (see [4] for a full description of the questions). The order of the blocks - and thus the feedback types - was counterbalanced between the participants.

After completing the main experiment, a control experiment was run to test the participant’s baseline response times to auditory and haptic stimuli. In two sequences, participants were asked to press a key as soon as they perceived an auditory (450 Hz beep) or haptic (Senseglove vibration on thumb and index finger) stimulus. In each sequence, a single feedback type was presented 10 times in 0.5 s long stimuli, with random intervals (range: 1–5 s) between the previous key press and the presentation of the next stimulus. The full experiment never exceeded 1 h, and participants could take a break between blocks.

2.5 Data Analysis

To test our hypothesis, we looked at the differences in the grip force profiles between the different feedback conditions. To quantify grip force, we used the total of the force registered in the BioTac thumb and index finger tips that was normal to the object’s surface. For each trial, all signals were resampled 100 Hz, and the time at which the slip force threshold was passed was set to time = 0. The squeeze force at time = 0 was subtracted from the signal to make responses comparable between trials. Resultant forces were low-pass filtered 10 Hz using a second-order Butterworth filter. Then, time traces were averaged across repetitions, resulting in one trace per participant-condition combination. For each of these traces, the first peak in the time series was determined, and the time and height of the peaks were used as our outcome variables. Finally, squeeze force traces were averaged across participants for visualization purposes.

Prior to averaging across repetitions, and outlier analysis was performed by removing trials in which no squeeze force peak could be found. In this way, we selected trials in which participants reacted according to instructions. This outlier analysis removed 54 trials, which was 22% of the total number of trials.

For the control experiment, the time between the start of the stimulus and the participant’s key press was calculated for each trial, and averaged across repetitions. To analyze the NASA TLX scores, data for each question were converted to a 0–100 score and averaged across participants.

Fig. 3.
figure 3

Squeeze forces (mean ± 1 standard error), as measured at the remote robot’s fingers. (a) Average time traces (solid lines) across participants. Time = 0 represents the time at which the slip threshold was passed. (b) and (c) Times and heights, respectively, of the squeeze force peaks, with one marker per participant.

Full size image

3 Results

In our main experiment, we compared the effect of haptic and auditory feedback about object slip on the participant’s squeeze behavior. The average time courses of the squeeze forces and the peak analyses are shown in Fig. 3. This figure suggest that feedback shifts the peak in squeeze force closer to initiation of slip (at time = 0). A repeated-measures ANOVA showed that there was indeed a significant effect of condition on peak timing (\(\mathrm{F}_{2,18} = 4.2, p= 0.032, \eta _{p}^{2} = 0.32\)). Bonferroni-corrected posthoc tests of peak timing showed a significant difference between auditory and no feedback (\(t = -2.9\), \(p=0.029\)), while neither auditory and haptic nor haptic and no feedback differed significantly (all \(t\le 1.5\), \(p\ge 0.48\)). For peak height, the repeated-measures ANOVA failed to find a significant difference between the conditions (\(\mathrm{F}_{2,18} = 0.90, p= 0.43, \eta _{p}^{2} = 0.090\)), so no posthocs were performed.

The control experiment showed that responses to auditory stimuli were 60 ms faster than responses to haptic stimuli, with haptic latency being \(456\pm 62\) ms (mean ± standard error), and auditory latency being \(396\pm 67\) ms. A paired Student’s t-test showed that this difference was significant (\(\mathrm{t}_8 = 2.8, p = 0.024\)).

The NASA TLX scores, shown in Fig. 4, illustrate that the task required medium cognitive and physical load from participants, and that participants were satisfied with their own performance. Per-question repeated measures ANOVAs only found that Physical effort was exactly on the edge of significance (\(\mathrm{F}_{2,26} = 3.4, p= 0.050, \eta _{p}^{2} = 0.21\)), but in Bonferroni-corrected posthoc testing no significant comparisons remained. None of the other questions revealed significant differences between conditions (all \(\mathrm{F}_{2,26}\le 2.5\), all \(p\ge 0.10\)).

Fig. 4.
figure 4

Results (mean ± 1 S.E.) from NASA TLX questionnaire, which was administered after each feedback block. Physical effort was the only question that showed a significant effect of condition, with no significant posthoc comparisons.

Full size image

4 Discussion

Auditory feedback helped participants to react significantly faster to slip, meaning that audio could be a useful warning signal for slip in tele-operation applications. While participants reacted faster, they did not significantly change their peak squeeze force. This shows that the auditory feedback did not induce a smaller safety margin, but also that users were still able to regulate their grip, while moving faster. Haptic vibration feedback failed to reach a significant effect on reaction time and squeeze force. This difference between the types of feedback could be caused by the small but significant (60 ms) difference in reaction times found in the control experiment. Another contributing factor could be the time it took the vibration motor to get to full vibration amplitude, which would add latency (\(\sim \)50 ms) to the onset latency which we already corrected for. Therefore, future experiments with lower-latency haptic devices are required to confirm the usefulness of haptic vibration feedback as a warning signal for slip.

Even though we do not see large improvements on NASA TLX scores with feedback, we also see no negative effects. Since the feedback was abstract, it could have distracted the users or placed an extra cognitive burden on them, but the current results suggests that the feedback was still fairly easy to process.

We had to discard a full data set and 22% of the trials of the remaining participants, because we encountered technical problems in various parts of the tele-operation system. Moreover, the BioTacs proved to be very susceptible to drift. All these issues probably contributed to the relatively large spread in our outcomes variables. Even though these were not ideal experimental conditions, we still were able to gather enough data to sketch an image of the usefulness of slip feedback. It also illustrates the complexity of full tele-operation systems, and thus underlines the need for testing feedback in these actual scenarios.

Future work on slip feedback could focus on developing a method to initiate slip more instantaneously, and with more control over its magnitude and duration. Such a method would allow for presenting realistic slip profiles, without providing additional visual cues. The slip detection procedure itself could be iterated on, by for instance adding information from the micro-vibration sensors in the BioTacs [1]. Another interesting avenue would be integrating more realistic, low latency slip feedback devices on the user’s side. Ultimately, these developments might allow tele-operation users to recruit reflexive behavior that is present in normal interactions with physical objects.