1 Introduction

Manipulation of objects in virtual reality (VR) commonly suffers from the absence of haptic sensations. As such, it is often unclear whether contact between one’s virtual hand and virtual objects has been made, whether an object is properly grasped or not, and what the physical properties of the hand-object contact are. Conventional haptic interfaces for VR, be they grounded  [9], body-grounded  [11], or handheld  [6] address this issue by applying forces to the user through an end-effector (e.g., a stylus), which mimics sensations of making and breaking contact as well as effects of mass, inertia, and collisions with the environment. However, such interactions are always mediated by the interface’s end-effector, degrading the experience and preventing simultaneous manipulation and exploration of the virtual object. Encounter-type haptic displays (ETHDs) solve the issue of rendering sensations of making and breaking contact, bringing their end-effector in contact with the user only when collisions with virtual objects occur  [16]. Many types of grounded  [7] and body-grounded ETHDs  [10] exist. However, to the best of our knowledge, very few tackle the issue of grasping and manipulating objects. One work in this direction is that of  [13], whose device allows grasping of the tangible end-effector, but presents the same issues as conventional haptic interfaces if the user wants to manipulate the grasped object. Passive haptics offers an alternative solution, superimposing virtual objects with similar tangible ones to create the illusion of truly manipulating virtual entities  [5]. However, the number of required props for passive haptics increases with the complexity of the scene, making this approach unmanageable in rich virtual environments. Several different approaches aiming at rendering multiple virtual objects with few tangible ones exist to address this issue. They use reconfigurable or active tangible objects  [4, 12], augment passive props via wearable haptics  [14, 15] or use redirection techniques  [1, 8].

This paper presents a novel solution called “WeATaViX” at the interface between ETHDs and passive haptics, in the form of a wearable encounter-type device whose end-effector is a tangible object. It aims to provide physical presence for virtual objects while remaining as simple and unobtrusive as possible. The device is grounded on the back of the hand, secured to the skin via an ergonomic adhesive silicone layer. A servo motor moves a rigid link equipped with the tangible object towards and away from the user’s palm. Unlike other wearable ETHDs (e.g., [3]), our end-effector aims at best fitting the shape properties of the virtual object, inherently solving shape rendering problems. With the device secured to the user’s hand, the relative placement between the tangible and user’s hand mimics that of their virtual counterparts. This paper presents our device, along with its dynamic analysis and a human-subject evaluation in VR.

2 The WeATaViX Haptic Interface

Fig. 1.
figure 1

Haptic device composed of a 3D-printed part anchored to an adhesive silicone layer attached to the hand. Two capacitive sensors cover the tangible, respectively facing the palm and the fingers during grasp closure.

Full size image

2.1 Design and Description

A prototype of the device is shown in Fig. 1. It is composed of a 3D-printed structure to be placed on the back of the hand. Its profile is slightly curved to fit the shape of the hand. On the internal side, it is anchored in an adhesive silicone skin based on work by Chossat et al. [2], guaranteeing good adherence, comfort, and adaptability to different hand morphologies and skin properties. A HTC Vive Tracker can be attached on the external side. The distal side of the 3D-printed structure houses a HiTec HS-5065MG servomotor which controls the motion of a rigid link holding the tangible object. By moving the rigid link, the motor brings the tangible object towards or away from the user’s palm. The tangible object is equipped with capacitive sensors to detect contacts with the hand. Further details are included in Fig. 1. A video of the device in action is available at https://youtu.be/JtcEYlwogpA. The device was designed with minimal weight as a target, weighing 85 g without the Vive tracker (185 g with tracker). Figure 2-A shows how the electronics are interconnected. Figure 2-B shows the VR setup. The HTC Vive tracker enables hand position tracking and, together with the capacitive sensors, animation of the user’s hand avatar in VR.

Fig. 2.
figure 2

(A) Schematic of the interconnected electronics structure for sensing and control. The capacitive sensing uses the Arduino CapacitiveSensor library. (B) VR setup.

Full size image

2.2 Evaluation of the Device Performance

Silicone Performance.    The skin-safe silicone layer allows good adhesion of the structure to the skin even when the servomotor is active and during fast hand movements. We observed that the device continues to adhere well even after prolonged use (>45 min) and throughout several attaching/detaching cycles (>30 cycles).

Interaction Delay. With the device mounted on a user’s hand, we measured the delay between the command to engage the tangible and the contact detection on the palm, starting either far from the hand (servo shaft rotation of 80\(^\circ \)), or very close (servo shaft rotation of 5\(^\circ \)) to contact (0\(^\circ \)). Over 100 trials, we measured a mean delay of 225 ms for the far position (SD 7.2 ms) and 49 ms for the near-grasp position (SD 8 ms). This leads us to estimate the fixed delay due to communication to be around 38 ms and the servo shaft rotation speed to be around the nominal rotation speed of 210 ms/90\(^\circ \) despite the tangible object.

Influence of Motor Vibrations. The motor’s motion sometimes induced transient vibrations of the Vive tracker which propagated to the hand avatar. To quantify this effect, a user wore the device with the palm facing downward against a fixed supporting structure. We recorded the tracker position while applying step motions to the servo shaft using the full range of movement of the tangible to maximise such vibrations. Over the course of 20 trials, we obtained a mean stabilisation time of 612 ms for the tracker, with induced positional errors up to 4.03 mm (SD 0.87 mm) and maximum angular errors of 2.76\(^\circ \) (SD 0.43\(^\circ \)).

Fig. 3.
figure 3

Invisible proxy and visible object are released at 0 ms and 10 ms with the current hand speed \(v_{hand}\). Their positions relative to the predefined grasping location in the virtual hand (orange) \(d_{prox}\), \(d_{vis}\) are used to compute a smooth command \(d_{cmd}\). (Color figure online)

Full size image

3 Interaction Technique in VR

We implemented the simplest functional interaction technique for our device, with the aim of evaluating what functionalities and limitations are thus incurred. The rendering of an interaction between the user’s hand and the tangible object uses a simple distance-based triggering paradigm. Whenever a virtual object comes close enough to the user’s hand, the motor’s shaft angle is driven proportionally to the virtual distance between the grasping location and the virtual object, moving the tangible object towards the user’s palm. When the user’s fingers touch the capacitive sensors through grasp closure, the object drifts to the predefined grasping location fitting a natural power grasp while the hand avatar is animated to envelop the virtual object. Upon release of the physical grasp, an invisible virtual proxy is released, followed by the virtual object 10 ms later. The tangible is immediately driven by smoothly interpolating the command position between that of the proxy and that of the virtual object over 100 ms (see Fig. 3). Although simple, this interaction technique elicited positive feedback from users.

4 User Study

We conducted a user study to evaluate our device and interaction technique in VR. We designed tasks covering a wide range of grasp and release interactions with different object speeds and positions relative to the user in order to determine the range of interactions supported by our device. 14 right-handed subjects (10 males, 4 females; ages 22–58 (M = 29)) participated in the study after providing written informed consent. Subjects wore the haptic device on their right hand, adjusted for their specific grasp. They viewed the virtual environment through a HTC Vive HMD, and held a Vive controller in their left hand to answer the experimental questions. We evaluated grasping and releasing in a static task where the virtual objects did not move, and in a dynamic task where the objects moved and had to be caught by the user. The virtual tasks lasted around 45 min per participant.

4.1 Static Task

Subjects stood facing a 10-cm-side cube (see Fig. 4-A) with the object appearing on one of its faces. They had to grasp and pick up the object using their dominant hand, after which they answered a first experimental question regarding synchronicity of the haptic and visual grasping interaction. They responded using a 5-point Likert scale ranging from “Not at all synchronous” (0) to “Totally synchronous” (5). In the second part of the interaction, they had to precisely place the object back onto a highlighted face of the cube. Upon releasing the object, they answered another question regarding synchronicity of the haptic and visual release using the same 5-point Likert scale. Trials were considered a failure if at any point, the subjects accidentally caused the object to drop. Subjects were instructed to minimize failures and task execution times. They began by performing 3 practice trials, then performed a total of 108 trials covering all combinations of grasping and releasing orientations with 3 repetitions each. After the trials, subjects filled out a questionnaire evaluating realism and ease of the interaction, device wearability and obtrusiveness, and task difficulty.

4.2 Dynamic Task

In the dynamic task, subjects were to catch virtual objects travelling at different speeds and arriving at different locations relative to their body. A cannon fired a single spherical object in a linear trajectory chosen amongst 7 options (see Fig. 4-B.2). The object travelled at one of three speeds: 1 m/s, 2 m/s or 3 m/s. Speeds and trajectories were chosen randomly such that an equal number of each speed was attempted for each trajectory and an equal number of attempts was made per trajectory. Subjects failed the trial if they failed to catch the sphere. After each catch, they were asked to rate synchronicity between the physical and virtual interaction, responding using the same 5-point Likert scale as in the static task. Subjects performed 3 practice trials, a total of 105 trials covering all combinations of object speeds and catching locations with 5 repetitions each, after which they filled out similar questionnaires to those from the static task.

Fig. 4.
figure 4

Static (A) and dynamic (B.1) task environments. (B.2) The 7 catching positions.

Full size image

4.3 Results and Discussion

Static Task.    There was no visible effect of picking or placing position for all metrics, indicating that our device allowed similar task performances regardless of configuration, despite a single physical object approach and release direction. Task times were measured between the moment subjects entered the interaction region to the moment they respectively left it with the object in hand or completed the placing task. Picking task times were consistent within subjects, but variable between subjects (M = 2.51 s, SD 1.98 s). Placing task times followed a similar pattern (M = 2.31 s, SD 2.56 s). These task completion times indicate the device allows picking and placing interactions in reasonably short times. We measured the time between user’s grasp closure and the contact between the tangible sphere and the palm of the hand. About half the population grasped the tangible object with the fingers first, while the other half waited for the physical object to collide with their palm for closing their grasp, which is an important consideration for the design of interaction techniques intended for assisting subjects during grasping. Subjects were consistently successful in both picking and placing tasks (M = 96.03%, SD 5.51% for picking; M = 86.24%, SD 9.04% for placing). Combined with the short task completion times, this is indicative of high adequacy of our device for grasping and releasing static virtual objects. We measured grasping positional error as the absolute distance separating the virtual object and the grasping position on the palm at the time of detected grasp closure. It appears all grasping orientations yielded similar errors (M = 2.6 cm, SD 1.1 cm). Subject’s evaluation of picking synchronicity appears to positively correlate with grasping positional errors. Overall, subjects rated picking interactions as synchronous (M = 3.91, SD 0.86) and placing interactions as even more synchronous (M = 4.17, SD 0.92), but not significantly. Grasped objects were perceived as realistic (M = 4.14, SD 0.66), again reflecting adequacy of the device to manipulating static virtual objects. Users overall felt only moderately free in their movements (M = 3.71, SD 0.82). They reported device weight, motor vibrations and wiring as sources of obtrusiveness, rating the device as only moderately unobtrusive (M = 2.71, SD 0.73). Subjects reported high perceived virtual hand ownership (M = 4, SD 0.55), indicating that even with very rudimentary animation of the virtual hand our system is capable of maintaining immersion. The task was reported as being moderately easy (M = 3.42, SD 0.94) and did not cause excessive fatigue to subjects on average (M = 2.64, SD 1.15), though this varied a lot from subject to subject. Subjects pointed out the disturbing device weight during prolonged use, making further weight reductions as a future design priority.

Fig. 5.
figure 5

Use case: (A) pick and throw an apple; (B) catch an apple falling from a branch.

Full size image

Dynamic Task. In this task, almost all subjects tended to close their fingers ahead of the contact between the tangible object and the palm, indicating that subjects adapt their behavior to the task. These adaptations should be taken into account when designing interaction techniques supporting a wide range of tasks. Caught objects were perceived as moderately unrealistic (M = 2.93, SD 1.07), far below the perceived realism in the static task (means significantly different, \(p<0.001\), 2-sampled t-test). This is to be expected as the catching task amplified the perceptual effects of delays in our system. Subjects again felt moderately free in their movements (M = 3.43, SD 1.09) and rated device obtrusiveness similarly to that in the static task (M = 2.57, SD 0.76). Subjects reported a lower perceived virtual hand ownership than in the static condition (M = 3.07, SD 1; means significantly different, \(p<0.01\), paired t-test). Device limitations in the dynamic task combined with task difficulty seem to negatively affect the capacity of the device to ensure immersion. This highlights the need for improving the interaction technique if dynamic tasks are to be executed. Subjects reported that even on failed tasks, when the ball hit the hand and bounced off, the tactile feedback felt realistic. However, they also reported perceived delays in the haptic feedback which complicated the task and led to low perceived realism. Finally, the device required users to adapt their catching strategy, leading them to perform unnatural movements. This highlights a limitation of the current device and interaction technique when interacting with fast moving virtual objects.

5 Use Case

To showcase the adaptability of our device to multiple interactible virtual objects as well as the freedom of movement it provides, we designed a use case in which the user can freely roam about a virtual orchard, picking apples from trees, the ground, or tables, catching them as they fall, and even using them as ammunition in a game of “knock the cans” (see Fig. 5).

6 Conclusion and Perspectives

We presented and evaluated a novel wearable haptic interface at the boundary between encounter-type displays and passive haptics. The device is grounded on the back of the hand thanks to an ergonomic adhesive silicone layer and uses a servomotor to bring a tangible object towards and away from a user’s hand. We describe a simple interaction technique for VR allowing users to naturally grasp, manipulate and release objects while receiving compelling haptic feedback. Our device provides a simple and effective solution for tangible interaction with multiple virtual objects in large workspaces, with high adaptability to virtual environments. By grounding the device on the back of the user’s hand, our system is unaffected by tracking issues incluencing conventional passive haptics. Furthermore, by mixing aspects from tangible haptics and encountered-type displays, our work opens perspectives towards ETHDs that provide the possibility of manipulation and object exploration through grasp closure. While our current solution only features a single fixed tangible, the ultimate goal will be to provide interactions with interchangeable or reconfigurable end-effectors in order to increase adaptability. Our device received positive feedback from users during its experimental validation, however several issues and limitations remain to be overcome. In the short term, it will be necessary to make our device at least partially wireless and more compact to increase portability and freedom of movement. Also, a reduction of the carried mass, introduction of mechanical damping elements between the servo and tracker, and improvements to the control law are avenues we wish to explore to overcome the issue of unwanted vibrations. Since in our current implementation the motor responds to the release of the grasp on the tangible object, the real object lags behind the virtual object. Our simple interaction technique compensates for this when interacting with static virtual objects but was shown to be less adequate for interactions with moving objects. We plan to explore both improvements to the control law as well as to the interaction technique (e.g. contact prediction) to make our device adaptable to a wider range of virtual interactions. Currently, the device only allows a single physical grasp position. Additional capacitive sensors should allow differentiation of grasps and thus a much wider range of object manipulations. Our interaction technique also only admits a single optimal virtual grasping location, thus providing various forms of grasping assistance to the user may improve usability. In the longer term, we wish to investigate using our device to apply force feedback towards or away from the hand, to simulate mass and inertia.