Keywords

1 Introduction

Robots are frequently deployed to perform tasks that are dull, dirty or dangerous. Especially in repetitive tasks, technical advances allow the application of autonomous systems. Despite advances in autonomy, autonomous robots will not be a viable option in all applications for the foreseeable future [1]. If the tasks of a robotic system are diverse, the environments unpredictable and the stakes of successfully performing the task are high, robots will not be able to carry out all necessary tasks with sufficient reliability without human involvement. The common solution in these use cases is teleoperation (e.g. [2, 3]). In teleoperation the task is performed by an operator controlling the robot remotely, typically in a master-slave setup.

Currently the control of teleoperated systems is often not very intuitive and the cognitive load during control of the system is high for the controller. One of the improvements is to give the operator good dexterity and haptic feedback (e.g. [4]. The most intuitive haptic feedback is to present the forces on the robot’s hands and fingers as forces on the operator’s hands and fingers. However, this requires complex actuators and limits wearability. An alternative method is to provide feedback through other sensory cues like vibratory or visual cues (sensory substitution). In this study, we compared the performance in a teleoperated Box & Block test without force feedback, with direct force feedback and with substituted force feedback in the form of vibration on the fingers. We choose the Box & Block test (e.g. [5]) following Catoire et al. [6] who described a teleoperation test battery with standard tests that can be used to benchmark system dexterity, and allow to advance the design, quantify possible improvements, and increase the effectiveness of the teleoperated system. The Box & Block test is one of these tests and focuses on dexterity.

2 Teleoperation Set-Up

In this study, a teleoperation setup consisting of a telemanipulator, a haptic control interface and a visual telepresence system was used (see Fig. 1). The telemanipulator consists of a four-digit, 13 degrees of freedom humanoid robotic hand (Shadow Hand Lite, Shadow Robot Company, London, UK), equipped with 3D force sensors (Optoforce OMD 10) on its fingertips, mounted on the flange of a KUKA IIWA 7 serial link robot with 7 degrees of freedom (KUKA Robotics, Augsburg, Germany).

Fig. 1.
figure 1

The teleoperation setup. A) the master side including a subject wearing the HMD, Senseglove and Elitac glove, and B) the slave side with the teleoperated robot arm and PTRU.

Full size image

The haptic control interface is realized by a haptic glove (Senseglove DK1, SenseGlove, Delft, the Netherlands) that tracks finger movements in 11 degrees of freedom and can provide passive force feedback on each of the fingers. Underneath the Senseglove a custom vibrotactile glove (Elitac, Utrecht, the Netherlands) was worn, which contains 16 strategically placed pancake motors. The movements of the hand in space are recorded by an HTC Vive tracker (HTC, Xindian, Taiwan) mounted on the Senseglove. The visual system is an in-house build, closed system that transfers the images from a stereocamera mounted on a pan-tilt-roll unit (PTRU) to a custom made Head-Mounted Display (HMD), which in turn controls the movement of the PTRU [7].

The software interfacing between the telemanipulator and haptic control interface was realized using two PC’s running the Robot Operating System (ROS), version Kinetic Kame on Ubuntu 16.04. The PC’s, one controlling the telemanipulator hardware and one controlling the haptic control interface, were connected through a local, dedicated gigabit ethernet network, with a typical latency of a few milliseconds.

2.1 Controls

The control of the robotic system was the same throughout the experiment. The KUKA arm is controlled via the fast research interface (FRI), in impedance mode. With a 1 kHz update frequency, the robot tracks the Cartesian end-effector setpoint with a translational stiffness of 150 N/m and a rotational stiffness of 100 Nm/rad. The null space motion is uncontrolled and therefore compliant. At the start of each run, the robot was returned to the same initial position.

The setpoint of the robot is determined by the position and orientation of the Vive tracker on the back of the hand of the operator, updated with 60 Hz. The operator switches control on and off using a foot pedal, using a common ‘clutching’ logic: as long as the pedal is pressed, the robot follows the movements of the user. When the pedal is released, the robot motions are decoupled and it remains at the current position. When the operator decides to continue control by pressing the pedal, the current position of the user’s hand is used as the reference position for new movements.

The robot hand is continually controlled by mapping the Senseglove measurements of finger positions to joint positions on the Shadow Hand at 50 Hz. This mapping was calibrated before the experiment by using two predefined hand postures (i.e. flat hand and strong fist) to obtain parameters that define the operator’s hand model, which was used in finger position estimation. A linear mapping between these finger positions and the joint positions of the robot fingers was empirically determined to reach a high degree of movement mimicking.

2.2 Haptics

In the experiment, the visual feedback of teleoperation could be accompanied by haptic feedback from the robot’s sensors to the operator. Two types of haptic feedback were used; braking force at the fingertips (direct force feedback; DFF) and vibrations at the fingertips (substituted force feedback; SFF). A third type of feedback involving vibrations at the back of the hand functioning as a binary signal indicating collisions was also measured in the experiment, but not analyzed for this paper.

For the DFF a fingertip was blocked when the Euclidean norm of the force vector measured at the robot finger exceeded 0.1 N. This mimics making contact with a non-compliant object. The SFF provided gradual feedback of the measured force at the robot finger in the form of vibratory feedback at the corresponding fingertip using the vibrotactile glove. A force with a norm of more than 0.05 N is linearly scaled to a maximum vibration at 2 N. The Elitac glove provides a logarithmic stimulation intensity in 16 steps. A pilot study has shown that the vibration steps are identifiable and judged linearly on a magnitude scale. Since DFF provides proprioceptive-information and SFF provides information on the applied amount of force, DFF and SFF were combined to explore potential additive effects of the feedback types.

3 Methods

3.1 Participants

Twenty-two participants (13 female, mean age 29.5 ± 10.0 yrs) participated in the three pretests of the experiment. Fourteen of them passed the third pre-test and were selected for the main experiment (10 female, mean age 25.6 ± 8.0 yrs). None of the participants had previous experience with teleoperation and all participants were naïve about the details of the experiment and were compensated for their time. All participants gave their written informed consent prior to the experiment. The setup and experiments were approved by the TNO Internal Review Board (registered under number 2019-026).

3.2 Experiment Design and Procedure

Adapted Box & Blocks Evaluation

For this study the classic Box & Block test was adapted to fit the capabilities and limitations of the visual part of the teleoperation setup. The depth of the start compartment was decreased to be able to more easily grasp the blocks (3.5 cm instead of the traditional 7.5 cm depth [5]). We also reduced the number of blocks in the test to 30 to give the participants a bit more space in the box, which made placing the (robot) fingers around the blocks easier (Fig. 2A).

Fig. 2.
figure 2

A) Adapted box & blocks task, B) Difference in completion time for the standard and adjusted box & blocks test.

Full size image

As in the original task, the aim is to move as many blocks, i.e. cubes sized 2.5 cm square, as possible from one side of the box to the other side while using one hand and it is not allowed to throw the block over the partition. The participant is seated while performing the test and the box is placed directly in front of the participant on a table. In our reference task, we asked participants to move 30 blocks as fast as possible with their own hand directly. We did this both from the higher side to the lower side and vice versa in a randomized fashion. This gave us an idea about the effects on performance due to the adjusted depth of the box.

Teleoperation Pre-test

The teleoperation pre-test consisted of a simple displacement task with blocks from the Box & Block test while using the teleoperation system without feedback. Four blocks were located in the middle of an A3 sheet of paper with four equal squares (9 cm) at the corners. The participants had to move the blocks to the predetermined squares, one to each corner. To keep control of the duration of the pre-test, the maximum time to do this was 4 min. Participants had three attempts, and had to move (at least in one try) all four blocks to the corners within 90 s to be included in the second subtest. This criterion was not known to the participants. Regardless of the result of the teleoperation pre-test, the participant was asked to do the adapted Box & Blocks again.

Main Experiment

The main experiment started with written instructions and a practice session performing a simple teleoperation task, i.e. moving several blocks without haptic feedback. Next, the different types of haptic feedback were explained and presented. Four types of haptic feedback were analyzed in this paper: No feedback (NF), Direct finger force feedback (DFF), Substitute finger force feedback (SFF), Both DFF and SFF. The participant performed the teleoperated (modified) Box & Blocks task in all conditions, each presented once in a semi-balanced (incomplete Latin Square) order. Between the conditions there was a short (ca. 2 min) break, in which the NASA-TLX questionnaire was answered. A 10 min break was scheduled after all conditions. This was repeated a second and third time, resulting in three repetitions per condition.

Teleoperated Adapted Box & Block Task

In the teleoperated Box & Block test, participants always moved the blocks from the higher side to the lower side. They were asked to move as many blocks as possible to the other side of the box within two minutes.

Subjective Measures

The NASA Task Load Index (NASA-TLX) was used to measure workload [8]. Furthermore, the participants were asked to rank the experimental conditions in order of preference.

3.3 Data Analysis

The data of the adapted Box & Blocks pre-test were used to quantify the effect of the adjustments that were made on the Box & Blocks test. The data of the teleoperation pre-test were used as exclusion criteria for the main experiment.

In the main experiment, the effect of haptic finger force feedback on task performance was compared to the performance without haptic force feedback. Additionally, a comparison between direct force feedback and substitute force feedback was made.

To investigate task performance in the teleoperated Box & Blocks task, we analyzed the number of blocks moved in two minutes. Furthermore, we analyzed the subjective scores on the NASA-TLX test, i.e. Mental load, Physical load, Time pressure, Effort, Performance and Frustration. Participants who moved 2 or fewer blocks in one of the repetitions (regardless of whether the haptic feedback type in which this occurred was evaluate in this paper or not) were excluded from the analysis. This occurred for two participants. Next, the data were analyzed with 2 × 2 × 3 RM ANOVAs (DFF × SFF × repetition) on all metrics. Greenhouse-Geisser corrections were applied when sphericity was violated. In these analyses, main effects would show the influence of either DFF, SFF and repetition on the performance, while interaction effects would be able to reveal a potential additive effect of combining DFF and SFF and potential differences in learning for the feedback types.

4 Results

Adapted Box & Blocks Evaluation

Participants needed on average 24.2 s (range 18.5–39.3 s) to move all 30 blocks from one side of the box to the other. A 2 × 2 RM ANOVA (height of box (low/high) × timing (before/after)) showed significant main effects of both the height of the box, F1.0,21.0 = 10.73, p = .004, and the timing F1.0,21.0 = 8.04, p = .010 (Fig. 2C). These effects show that participants were on average about 1.5 s (ca. 6%) faster starting at the adjusted (=higher) side of the box, and that participants were about 1.3 s (ca. 5%) faster in their second repetition.

Teleoperation Pre-test

Fourteen out of the 22 participants in the pre-test fulfilled the criteria to participate in the main experiment. From the eight that were not selected, two participants were not able to perform the task at all. The other six non-selected participants exceeded the 90 s time requirement (range 100–406 s).

Main Experiment

On average the participants moved 11.2 (range 3–21) blocks in the teleoperated Box & Blocks task. In the 2 × 2 × 3 RM ANOVA (DFF × SFF × Repetition) we found that neither forms of haptic feedback on the fingers (DFF, SFF and DFF + SFF) improved the performance on the number of blocks since there were no significant main effects and no significant interaction between DFF and SFF (statistical details can be found in Table 1). It is interesting that all except one participant had a preference for one of the feedback types over NF. For DFF, 5 out of 12 participants ranked DFF over NF, however the NASA-TLX results showed that DFF scored worse than NF on several aspects (time pressure, performance and frustration). SFF was ranked higher than NF by 8 out of 12 participants, and both objective and subjective measures were similar to no feedback. These results suggest that people preferred feedback even when it did not improve performance in our task. When comparing the ranking of the different feedback types, DFF, SFF and DFF+SFF, were ranked higher than the other feedback types by respectively 4, 5, and 3 out of 12 participants. This shows that the preferences for the feedback types are highly individual, and that more feedback (DFF + SFF) is not systematically preferred.

Table 1. Statistical design of the main experiment
Full size table

Furthermore, for number of blocks and effort main effects of repetitions were found (respectively p = .004 and p = .045), indicating a better performance (i.e. more blocks and less effort) in the later repetitions. This shows a learning effect in the teleoperation task of ca. 8% per repetition (Table 2).

Table 2. Results of the 2 × 2 × 3 RM ANOVA’s (DFF × SFF × Repetition)
Full size table

5 Discussion and Conclusion

In this study we compared the effect of different haptic feedback types on performance in a teleoperated Box & Blocks task. No difference was found in objective performance in the conditions with or without haptic feedback, but all except one participant had a preference for one of the feedback conditions compared to no feedback. Comparing direct force feedback and substituted vibration feedback also no difference was found in objective performance, but there were differences in preference among the participants. The lack of results on objective performance might be caused by the level of difficulty and lack of sensitivity of the teleoperated Box & Blocks task; to get a significantly higher score one additional block should be moved, which corresponds to an increase of about 8%.

In all conditions participants were able to perform from the first trial, but still performance increased over repetitions. This shows that our set-up is intuitive, but that there is learning involved. The improvement over repetitions in the teleoperated Box & Blocks task was similar in size as the improvement over repetitions in the Adapted Box & Blocks pre-test (ca. 8% and 5% respectively), which shows that the learning had to do mainly with the task and dexterous manipulations itself and not that much with the teleoperation set-up or the haptic feedback. Moreover, there is no interaction between the repetitions and the type of feedback. This indicates that first shot performance and learning with SFF is as good as with DFF. Whether performance would change when learning is finished or whether the maximum performance with the different feedback types is on the same level is impossible to conclude from this study, but would be very interesting for future research.

At first sight it seems surprising that there were higher scores on frustration and time pressure for the direct force feedback compared to no feedback. However, this might be caused by the design of the direct force feedback; to be able to feel hard surfaces the feedback blocked the fingers almost immediately when force was applied at the fingertips of the robot hand. This resulted in some unfortunate situations in which the blocks were not firmly hold yet, but the fingers could not close further. Future research is planned on the effect of haptic feedback when the visual information is less reliable or on more subtle dexterous tasks and feedback.

To conclude, haptic feedback is preferred in teleoperation, the Box & Blocks task is a too coarse test for our current teleoperation set-up, and vibration feedback as substitute for direct force feedback works well and can be used intuitively.