1 Introduction

Providing haptic feedback in virtual reality (VR) improves the user experience and performance (Aguerreche et al. 2010; Azmandian et al. 2016; Cheng et al. 2015; Hoffman 1998; Insko 2001; Sra et al. 2016). Some solutions for haptic feedback use what is known as "Active Haptic Feedback" (AHF), employing computer-controlled actuators to provide haptic feedback (Zenner and Kruger 2017). This approach enables the generation of tactile or kinesthetic sensations on the user's skin, allowing to explore the environment and understand its constraints (Kyriakou and Hermon 2019), for example wearing vibrotactile gloves or through robots (McNeely 2013). However, these devices are often expensive and interfere with the user experience. On the other hand, "Passive Haptic Feedback" works by mapping virtual objects to static physical objects with a similar shape in the real world (Insko 2001), and the real objects are also known as “props”. This approach usually improves the user experience (Hoffman 1998) without adding interfering or expensive devices. However, requiring a physical prop for each virtual object or for each shape is not always practical. Situations where it is not possible to use a physical object to provide haptics include in situ and mobile scenarios "on the street", "out of the office" or "in the wild" for casual or spontaneous use of virtual reality application in those situation physical props may not be available.

We explored practical and scalable alternatives to represent multiple virtual objects using limited resources. We drew inspiration from “the theater of the poor” movement by Jerzy Grotowski (Grotowski et al. 1967), who represented objects using their own body as a prop instead of real object props. This practice led us to define the Hand-as-a-Prop technique. In this technique, the hand acts as a traditional prop and provides haptic feedback. Following this strategy, we designed and implemented two versions of the technique: Self Hand-as-a-Prop and External Hand-as-a-Prop. For example, using the Self Hand-as-a-Prop in an object translation task, would work as follows: The user's non-dominant hand is moved to the position of the virtual object to be manipulated and tries to get its shape; meanwhile, with the dominant hand the user grabs, moves, and releases it. In this way, users benefit from the haptic feedback provided by themselves, a solution that does not require any external element to emulate the presence of virtual objects.

In this study, we aim to determine what the effect on User Experience and Performance of Hand-as-a-Prop is. We specifically investigate how object representation and manipulation might be affected. The research questions are as follows:

  • Is it possible to use human hands (either the user hand or an external person hand) to provide haptic feedback in Virtual Reality?

  • Is it possible to represent different shapes?

  • What is the effect in an object manipulation task?

The results of the conducted user study show that using hands as props is feasible in both modalities: self and external. In all the cases, users were able to accomplish the translation task with acceptable performance. Self Hand-as-a-Prop provided better performance than External Hand-as-a-Prop, sometimes with similar values to Object Props. Regarding User Experience, both Hand-as-a-Prop techniques were rated neutral tending to positive. Finally, users preferred using their hands rather than external hands.

In summary, the contributions of this work are as follows:

  • The design and implementation of Hand-as-a-Prop technique for self and external modalities.

  • A study on the ability to represent different object shapes.

  • A comparison of both techniques to real object props in a translation task.

2 Related work

The related work is classified into three categories: (1) VR Haptic Feedback Devices split into Active and Passive, (2) Visuo-haptic illusions based on visual dominance and proprioception applied to static and dynamic props, and (3) Self-haptics approaches where the user’s body is employed for physical feedback.

2.1 VR active and passive haptic feedback

2.1.1 Active haptic feedback (AHF)

One of the most common approach to provide haptics in VR is through AHF devices. Wearable gloves such as Wolverine (Choi et al. 2016) can provide tactile and force feedback. Users wearing vibrotactile gloves can feel textures and forces, but they can still push and clip through the objects, which reduces presence and immersion. Externally grounded devices, like Phantom (Massie et al. 1994), address this issue and provide a sensation of stiffness of the virtual objects, avoiding clipping through them. However, in large-scale systems, the approach is intrusive and requires bulky and expensive devices.

Shape-changing displays are classified as encountered-type haptic devices since they present a contact surface to the user’s hand (Abtahi and Follmer 2018). They provide AHF by dynamically changing their shape to represent virtual objects. However, this solution still presents limitations such as their cost, speed, size, and low spatial resolution (Abtahi and Follmer 2018).

2.1.2 Passive haptic feedback (PHF)

Previous research indicates that passive haptics enhances the sense of presence and immersion in virtual reality (Azmandian et al. 2016; Sra et al. 2016). PHF commonly employs static physical objects called props, usually using a one-to-one mapping where each virtual object is associated with a prop.

Handheld props are the most common approach, i.e., physical objects with a similar shape to virtual objects. One of the first applications was using a doll’s head for 3D neurosurgical visualizations (Hinckley et al. 1994). The use of traditional static props presents two main limitations: it is not scalable since too many props might be required for mapping each virtual object and requiring a placement mechanism since something or someone has to place the correct prop in the target place at a specific moment. Mapping each virtual object to a physical prop is neither scalable nor flexible.

Several placing and switching strategies of props have been explored. Robots can accomplish this task, for instance, Robotic Shape Displays (Araujo et al. 2016; McNeely 2013) use robotic arms to place props in front of the user's hand when they are meant to be reached. There are attempts to use non-grounded devices like drones to provide positional physical feedback (Hoppe et al. 2018; Knierim et al. 2017, 2018). For an extended review of encountered-type Haptic Feedback on Demand see (Mercado et al. 2021).

2.2 Visuo-haptic illusions

Vision dominates over other senses when they are in conflict (Colman 2015; Gibson 1933; Hecht and Reiner 2009). In addition, vision dominates over proprioception (Burns et al. 2007) which is the sense that allows perceiving us the location, movement and actions of parts of our body (Taylor 2009). This can be used to simplify the tactile stimuli necessary to represent certain virtual entities. Techniques such as Redirected touch and haptic retargeting exploit visual dominance effect.

Redirected touch enables a one-to-many mapping, making it possible to use the same prop to provide haptic feedback for multiple virtual objects. Kohli et al (2012, 2013) accomplished it by warping the visual virtual environment to direct the user hand toward the same physical prop, even when in the virtual world the hand seemed to be reaching different objects. They compared one-finger interaction in warped and non-warped environments, measuring performance using the Fitts’ law for a multidirectional tapping task, concluding that users can effectively interact in a warped virtual space. In (Abtahi and Follmer 2018), redirected touch is used to increase the spatial resolution of shape displays, and in (Zhao and Follmer 2018), redirected touch was extended for multi-finger interaction.

Azmandian et al. (Azmandian et al. 2016) propose the Haptic Retargeting framework to make statics props more flexible and scalable. Haptic retargeting defines a dynamic and minimally noticeable mapping that allows reusing physical objects to provide haptic feedback. This is possible by leveraging the visual dominance and dynamically aligning physical and virtual objects while the user interactions in the environment. Similarly, the Sparse Haptic Proxy (Cheng et al. 2017a) technique makes use of a single human-size physical proxy which provides haptic feedback for many virtual objects. However, in both cases, external devices are needed, the interaction target has to be known beforehand and it only works when the movement starts with the hand resting on the initial position.

Finally, shape-changing devices also take advantage of the visual dominance effect. The approach by (Abtahi and Follmer 2018) improves perceived speed and spatial resolution by using redirection, scaling, and retargeting visuo-haptic illusions, this research is limited to one-finger interaction and results are dependent on the size and resolution of a specific shape display. The experiment by (McClelland et al. 2017) evaluated how a shape-changing device approximates virtual object shapes.

2.3 Human-provided feedback

Human actuation has been investigated by Cheng et al. (2014, 2015, 2018); Cheng and Marwecki et al. (2017) without users being aware that another human is on the other side. In Haptic Turk (Cheng et al. 2014) and TurkDeck (Cheng et al. 2015), dedicated humans presented and operated the props: walls, tables, chairs, and steps. Thus, physical representations are created on the fly, enabling the creation of arbitrarily large virtual worlds. Mutual Human Actuation (Cheng and Marwecki et al. 2017) uses human feedback, but in this case, all users are immersed in different virtual environments. The iTurk technique (Cheng et al. 2018) does not require an extra human, but it is the user who provides forces to object props in his own virtual reality world.

Providing self-haptic feedback has been previously explored: Koli and Whitton (2005) designed “the haptic hand”, a system that uses the non-dominant hand to provide haptic feedback to the dominant one. They explored the use of the non-dominant hand as touch surfaces, for example simulating a panel where the user had to interact with the dominant hand. Despite obtaining promising usability results, a formal comparison with alternative techniques was not conducted. Also, more complex object representations or tasks such as translating objects were not explored. (Ban et al. 2015) applied retargeting to create the sensation of haptic feedback in one-hand tasks, they warped the index and thumb fingers so when the user was pinching an object they see the fingers in contact with the object’s surface, but in reality, the fingers are touching each other. Similarly, (Bovet et al. 2018) applied retargeting techniques to the locations of the user’s hands in order to make them act as props for the other hand. Evaluations indicate that the technique can effectively provide touch, and when compared to no-haptic feedback interaction, it improves immersion and realism. Recently, Fang and Harrison self-haptic technique (2021) was not compared to alternative haptic mechanisms such as traditional object props or other person hand and was not measured in a manipulation task as we do, though Fang and Harrison implemented some two-handed tasks, whereas we only implemented one-hand manipulation of a single object.

To summarize, haptic feedback techniques have been explored to be more practical, flexible and improve the user experience in virtual reality environments. Shape-changing devices (McClelland et al. 2017) can represent several objects with a single device. Also, previous research (Cheng et al. 2014, 2015, 2018; Cheng, Marwecki et al. 2017) used human agents and users’ own force to animate objects that will provide haptic feedback. Self-haptic feedback approaches exploiting the non-dominant hand as props have also been explored (Fang and Harrison 2021; Kohli and Whitton 2005). In our work, we explored two novel techniques to provide haptic feedback in virtual reality without needing a physical object (Self Hand-as-a-Prop and External Hand-as-a-Prop). In addition, we compared them with a traditional object props technique. In particular, Self Hand-as-a-Prop does not require an external agent but is the own user who provides haptic feedback himself.

3 Design and implementation of hand-as-a-prop

Hand-as-a-Prop takes inspiration from artistic performances. In theater, the human body is used on stage as a wall, table or handheld item; this was popularized by the “theater of the poor” movement by Jerzy Grotowski (Grotowski et al. 1967). Conceptual art borrowed the idea and applied it in performances such as “Rent-a-Body” (Cao 1993).

Hand-as-a-Prop is a haptic feedback technique that does not require an active or external device to represent virtual objects; it only requires the hands. We designed the technique to take advantage of the visual dominance effect to convey the illusion of perceiving haptic feedback from the virtual objects. We also expected that proprioception could help to locate the object when the own hand is used is used as a prop. Furthermore, as the hand can adopt multiple shapes similarly to shape-changing devices (McClelland et al. 2017), the technique allows modeling several objects using the hand. There are two modalities of Hand-as-a-Prop technique: External and Self.

In External Hand-as-a-Prop, the hand of an external person models the shape of the virtual object and it is the external person who places the hand to match the virtual object position, as seen in Fig. 1 External Hand-as-a-Prop. This technique is based on Haptic Turk, TurkDeck, and Mutual Human actuation techniques by Cheng et al. (2015, 2017) as an external agent provides actuation; however, we do not use any real prop. External Hand-as-a-Prop shares some of their disadvantages: the need of an external human agent, the agent has to be aware of virtual reality positions and actions, and they have to let their body to be moved by the user.

Fig. 1
figure 1

Hand-as-a-Prop and traditional Object Prop techniques. First row: the virtual environment. Second row: External hand-as-a-prop: an external person places his/her hand following instructions from an external monitor. Third row: Self Hand-as-a-Prop: the user positions his/her hand on the virtual object position. Fourth row: Object Props, an external human has to visualize in a monitor where to place the corresponding real object

Full size image

In the Self Hand-as-a-Prop case, it is the user who is in charge of modeling the shape of the target object in the correct location to match the virtual object position, see Fig. 1 Self Hand-as-a-Prop. There is no need for an external display to show the position and shape of virtual objects. A hand pose algorithm is employed to detect the hand gesture and determine if it matches the target virtual object. When this occurs, the system enables the manipulation of the virtual object.

3.1 Implementation

To implement the Hand-as-a-Prop techniques we used a head-mounted display with a hand tracking system, the Oculus Quest 2 virtual reality headset. Both user’s hands are displayed in virtual reality, except when the hand is representing a virtual object in the Self Hand-as-a-Prop technique. The hands models are from the Unity Oculus Virtual Reality Integration Package. Only the dominant hand had the possibility to grab.

3.1.1 External hand-as-a-prop implementation

In this case, an external human places his/her hand in the position and shape of the virtual object. The hand tracking system incorporated in the head-mounted display is used to let the user see his real non-dominant hand position. The external agent hands were not tracked, and the virtual reality scene visualization is duplicated in an external display so that the external agent is aware of the target shape and position for the hand (Fig. 1 External Hand-as-a-Prop). The external agent has to place his/her hand before the user approaches and grabs it. During the manipulation, (Fig. 3 External Hand-as-a-Prop), the external agent has to let his/her hand be moved and translated freely.

3.1.2 Self hand-as-a-prop implementation

Initially, the user non-dominant hand is visualized in red and the user performs the matching gesture for the specific virtual object as seen in Fig. 2 Self Hand-as-a-Prop. We used a static hand gesture detection method based on the distance between the joints to three pre-defined gesture poses. Upon detection of a gesture, the hand transforms into the object, meaning that it is ready to be grabbed as seen in Fig. 2 Virtual Object. When the detected hand is grabbed, it cannot be tracked because it is being occluded by the dominant hand (Fig. 3 External Hand-as-a-Prop). At any time, if the grabbed hand becomes visible, it means that the object is not being grabbed anymore, and therefore, the virtual object is released.

Fig. 2
figure 2

Objects of different shape: cube, cylinder and flat volume. Real hand gestures (closed fist, thumb up, and flat palm), virtual hands recognized in Self Hand-as-a-Prop, and real objects to be used in the baseline Object Props condition

Full size image
Fig. 3
figure 3

Manipulation of a virtual object: first row shows the object placement in virtual reality, a virtual hand approaching the object and the hand grabbing the object; second row with the External Hand-as-a-Prop technique, third row with the Self Hand-as-a-Prop technique, and fourth row manipulation with a real object prop

Full size image

4 User study

In this section, we evaluate the user experience and performance of the technique in a virtual reality manipulation task in order to answer our research questions: Is it possible to use human hands (either the user hand or an external person hand) to provide haptic feedback in Virtual Reality? Is it possible to represent different shapes? What is the effect in an object manipulation task?

To investigate these questions, we designed a within-subjects user study consisting of two different tasks; see Fig. 4. The first task (Exploration task) was to statically assess the goodness and realism of the physical representation of the objects. This task is based on experiments performed by McClelland et al. (McClelland et al. 2017) and Wobbrock et al. (Wobbrock et al. 2009) aimed at evaluating shape-changing haptic devices. This task was designed to evaluate object shape and haptic condition. Selected shapes were a cube, a cylinder, and flat volume, Fig. 2 shows the virtual objects and their corresponding real objects that were 3D printed. Haptic conditions were External Hand-as-a-Prop (E_H), Self Hand-as-a-Prop (S_H), and traditional haptic feedback based on object props (O_P). In the Object Props case, an external human agent is employed to place the objects in the position of the virtual object.

Fig. 4
figure 4

Experimental design

Full size image

The second task (translation task) was only aimed to evaluate the haptic condition. We evaluated the task completion time (TCT) manipulation time (MT), fatigue, realism, and goodness of the representation during a pick and place task. We follow guidelines from a recent research (Bergström et al. 2021) on the evaluation of virtual reality manipulation tasks “Define the goal of the evaluation: Choose speed or accuracy”: we decided to evaluate task completion time because in preliminary evaluations there were noticeable differences in that metric that made us investigate further.

4.1 Participants

We conducted the experiment with 12 participants, which aligns with the sample size commonly used in human–computer interaction research with similar study designs, as recommended by previous works such as MacKenzie (2013) and Arif (2021). This approach was adopted because prior knowledge of the variance within a sample was not available, making it impractical to perform a priori power analysis for determining sample sizes.

Participants were 5 females, and 7 males between 20 and 45 years old, only two participants had not tried virtual reality before. Participants did not report motion sickness or any negative effect from virtual reality. Each participant performed the experiment individually and it took approximately 40 min per participant. After the exploration task and prior to performing the manipulation task, every participant was given a brief introduction for each haptic condition and had one minute to test the manipulation task with the corresponding haptic technique.

4.2 Procedure

We created a test environment with a virtual table, three objects (cylinder, sphere, and flat surface), and 6 areas where the objects should be placed during the translation task. Figure 1 shows the virtual table, a virtual cube, and a target area. The virtual world was aligned so that the virtual table matched a real table (90 cm * 68 cm), the virtual and real objects also matched (cube = 7 cm side, cylinder = 5 cm diameter, 15 cm height, flat = 3,5 × 7 × 15 cm). Target areas were 6 cm squares, they were arranged in a 2 × 3 grid, separated by 21 cm vertically and 27 cm horizontally. Participants performed the tests sitting in an adjustable chair. A head-mounted display was worn by the participants and they were instructed to remain seated during the experiment, but they still could change their point of view and move their torso and head. All the objects were within reach.

4.2.1 Exploration task

During each trial, one of the three virtual objects appeared on the table in front of the participants, where they could easily reach it with both hands. They were instructed to touch and explore the object. Once they explored the object, they were asked to answer two questions about the realism and the goodness of the representation of what they have physically explored. For goodness participants rated the statement “the virtual [object] is represented adequately by the shape of what I just explored with my dominant hand”, and for realism they rated the statement “the virtual [object] I just explored feels real”. Under the Self Hand-as-a-Prop condition, participants were also asked for the easiness of performing the gesture. In this case they rated the statement “the shape of the virtual [object] is easy to perform with my non-dominant hand”. The [object] was a Block, Cylinder or Flat. Each question used a Likert scale from −3 to 3, participants answered orally to avoid removing the headset in each trial. There was only one exploration of each shape by every user. Each user conducted 9 exploration tasks (3 conditions × 3 shapes), and this resulted in 108 trials.

4.2.2 Translation task

After the exploration task, the translation task started. During this task, one object appears in a pseudo-randomized area on the table, and in total, six areas were pre-defined on the table; for each condition 24 trials were performed, order and positions of objects are pseudo-randomized, but each shape appears 8 times. Positions and order are the same for all users. The participant had to grab and place the object on the target area, which is represented as a white rectangle on the table. Once they reach the target area, the contour of the object changes its color for 0.5 s and then stars emerge indicating that the trial has been completed. The first three trials were to warm-up and discarded from the analysis. At the end of the translation task, participants were asked to answer 8 questions about the easiness and realism of the translation of the objects, the fatigue of their dominant/non-dominant hands and arms, and two open questions where they could make general comments related to the objects’ translation technique and the fatigue. Each question used a Likert scale from 1 to 7 or a textbox when it was an open question. Participants answered the questions using a laptop. We collected 864 trials = 12 participants × 3 Conditions × 24 trials.

4.3 Metrics

For the exploration task, participants were asked about the realism (i.e., “the explored object feels real”) and representation goodness (i.e., “the explored object is correctly represented by what you just explored with your hand”) of what they have physically explored. Under the Self Hand-as-a-Prop condition, they were also asked about the easiness (i.e., “the gesture was easy to perform”) of performing the gesture (see Table 1).

Table 1 Measures of dependent variables
Full size table

For the translation task, the software measured full completion time and manipulation time. Additionally, after each block, participants were asked to complete a questionnaire assessing realism (i.e., “the translation of the objects feels real”), fatigue (i.e., assess how tired are the left and right arms and hands after the translation task), and easiness (i.e., “the translation of the objects was easy to perform”) of the translation technique. Lastly, they were asked to rank the conditions from 1st to 3rd regarding their preference (see Table 1).

5 Results and analysis

5.1 Task 1: exploration task

Two-way repeated-measures ANOVA was conducted to compare the effect of the three haptic feedback conditions (Self Hand-as-a-Prop S_H, External Hand-as-a-Prop E_H, Object Props O_P) and the different object shapes (Cylinder, Cube and Flat). We checked assumptions for ANOVA with Shapiro–Wilk’s test for normality and Mauchly’s test for sphericity. Mauchly’s tests indicate a violation for variables Representation Goodness and Gesture Ease and therefore Greenhouse–Geisser adjustment were applied. F, p, and η 2 values of statistical significance are reported in Table 2.

Table 2 Statistical measures of the exploration task
Full size table

5.1.1 Perceived realism

The ANOVA results showed that the main effect of the haptic feedback condition was statistically significant (F(2, 22) = 32.435, p < 0.001, η2 = 0.6) as well as the effect of the represented shapes (F(2, 22) = 9.874, p < 0.001). There was also a significant interaction effect between the haptic feedback condition and the represented shapes (F(4, 44) = 3.713, p < 0.05).

Regarding the haptic feedback condition, Post-hoc comparisons using Bonferroni corrections showed significant differences between E_H and O_P (p < 0.001) as well as between S_H and O_P (p < 0.001), but not between E_H and S_H.

The perceived realism in O_P is 2.6, corresponding to a high realism perception, while in S_H is 0.2 and in E_H is 0.3 (see Fig. 5), both slightly over the middle point of the scale, suggesting a neutral realism perception for both conditions of Hand-as-a-Prop. O_P is perceived as the most realistic representation of virtual objects. The perceived representation of realism is similar for S_H (M = 0.2) and E_H (M = 0.3), suggesting that the perception of realism might not be affected by the sense of touch from your hand, but by the fact of touching a hand.

Fig. 5
figure 5

Box plots for the results of the Exploration task questionnaires. Horizontal bars represent the median, the cross is the mean, and boxes represent the interquartile ranges (IQRs). Whiskers stretch to the data points that are within the median +−1.5IQR

Full size image

5.1.2 Representation goodness

Mauchly’s test of sphericity indicates a violation and therefore Greenhouse–Geisser adjustment was applied. The ANOVA results showed a significant main effect of the Feedback Condition (F(1.353, 14.887) = 33.774, p < 0.001, η2 = 0.565). There is also a significant main effect of the represented shapes (F(2, 22) = 7.785, p < 0.05) and an interaction effect Feedback Condition * object Shape (F(4, 44) = 3.526, p < 0.05).

The average representation of goodness for all shapes of the O_P condition is 2.56; it is significantly higher than E_H = 0.33 and S_H = 0.44 (see Fig. 5). Both E_H and S_H are slightly higher than 0, suggesting a neutral-to-positive perception of goodness.

The representation goodness is similar for S_H (M = 0.33) and E_H (M = 0.44), suggesting that both Hand-as-a-Prop variations are equally good whether they are external or the self.

5.1.3 Objects shape

Regarding the object shapes, results show that the cylinder is perceived as the least realistic in S_H (M = −0.5, SD = 1.62) and E_H (M = −0.5, SD = 1.78), but not for the O_P (M = 2.58, SD = 0.76) where it had similar perceived realism as the other physical objects. Similarly, the cylinder representation goodness was considered the worst in the S_H (M = −0.25, SD = 1.865) and E_H (M = −0.33, SD = 1.723) but not in the O_P (M = 2.67, SD = 0.49) condition where it has a similar score to the other objects. Values slightly under the middle point of the scale suggest a neutral-to-negative rate. We interpret such results as a consequence of the shaping capabilities of the human hand which makes it more suitable to mimic a cube and a flat surface than a cylinder.

5.1.4 Gesture ease

Mauchly’s test of sphericity indicates a violation and therefore Greenhouse–Geisser adjustment was applied. The ANOVA results showed that there is no significant main effect of the represented object (F(1.185,13.036) = 2.437, p = 0.14, η2 = 0.181); however effect size is not very high (Cohen, 1988). Thus, we cannot conclude that the three designed gestures were equally easy to perform. (Cylinder: M = 2, SD = 1.54, Cube: M = 2.58, SD = 0.79, Flat: M = 2.25, SD = 1.22).

5.2 Task 2: translation task

Repeated-measures ANOVA was employed to compare the effect of the 3 conditions (Self Hand-as-a-Prop S_H, External Hand-as-a-Prop E_H, Object Prop O_P). We report on objective and subjective measurements. Outliers were filtered out (i.e., mean ± 2 standard deviation). Shapiro–Wilk’s test results indicate data was normally distributed and Mauchly’s test indicate sphericity is met. Post-hoc comparisons used Bonferroni corrections. F, p, and η2 values of statistical significance are reported in Table 3.

Table 3 Statistical measures of the translation task
Full size table

5.2.1 Time

We measured the full task completion time (Full TCT) and manipulation time (MT) at the translation task (see Fig. 6).

Fig. 6
figure 6

Manipulation and Full Task Completion Time (seconds) split by condition (Self Hand-as-a-Prop S_H, External Hand-as-a-Prop E_H, Object Prop O_P). Error bars represent the standard error

Full size image
5.2.1.1 Full TCT

Results showed a significance on TCT (F(2, 22) = 8.415, p = 0.002, η2 = 0.433). The S_H (M = 4.61 s, SD = 1.03) and E_H (M = 4.56 s, SD = 0.82) conditions are slower than O_P (M = 3.9 s, SD = 1.03). For Full TCT, S_H and E_H present similar results. Thus, S_H is slower during the initial stage of each trial, i.e., from when the object appears on the table until it is actually taken with the dominant hand. This could be a direct consequence of the extra time from the gesture recognizer in the S_H condition.

5.2.1.2 Manipulation time (MT)

Results showed a significant effect on MT (F(2, 22) = 42.507, p < 0.001, η2 = 0.794) with differences between E_H (M = 2.66 s, SD = 0.36) and O_P (M = 1.83 s, SD = 0.42), but not between S_H (M = 2.04 s, SD = 0.488) and O_P. S_H was also significantly faster than E_H (p < 0.001). Interestingly, there was no significant difference between S_H and O_P. Regarding MT, results suggest that S_H and O_P conditions behave in a similar way, both faster than E_H. This indicates that moving someone else’s hand implies more time than moving your own hand. Representing the object with your hand seems to be helpful in terms of coordination, i.e., the non-dominant hand goes along with the dominant one assisting the translation movement.

5.3 Realism

Results showed a significant effect on the question “virtual object translation feels real” (F(2, 22) = 8.134, p = 0.002, η2 = 0.425), depending on the type of haptic feedback. Post-hoc comparisons using Bonferroni corrections showed significant differences between O_P (M = 2.25, SD = 0.75) and S_H (M = 0.5, SD = 1.68) or E_H (M = 0.67, SD = 1.5) (see Fig. 7).

Fig. 7
figure 7

Box plots for the results of the Translation task questionnaires. Horizontal bars represent the median, the cross is the mean, and boxes represent the interquartile ranges (IQRs). Whiskers stretch to the data points that are within the median +−1.5IQR

Full size image

The perceived realism for the translation under the O_P condition is 2.25, corresponding to a high realism perception, while the average perceived realism in S_H is 0.5 and in E_H is 0.67, suggesting a neutral-to-positive realism perception for the overall Hand-as-a-Prop. This confirms that O_P is perceived as the most realistic translation of virtual objects. The perceived translation realism is similar for S_H and E_H, suggesting that also during the translation of the objects, the perception of realism might not be affected by the sense of touch from the own hand.

5.4 Translation ease

Results showed a significant effect on the perceived realism question “virtual object translation was easy to perform” (F(2, 22) = 6.937, p = 0.005, η2 = 0.387), depending on the type of haptic feedback. Post-hoc comparisons using Bonferroni corrections showed significant differences between S_H (M = 1.67, SD = 1.23) and O_P (M = 2.58, SD = 0.52), and even stronger significant effects between E_H (M = 1.33, SD = 0.99) and O_P. The feedback condition evaluated as the easiest was O_P, then S_H, and lastly E_H (see Fig. 8).

Fig. 8
figure 8

Box plots for the Fatigue questionnaires. Horizontal bars represent the median, the cross is the mean, and boxes represent the interquartile ranges (IQRs). Whiskers stretch to the data points that are within the median +−1.5IQR

Full size image

The average perceived ease for the translation under the O_P condition is 2.58, corresponding to a high easiness. For S_H is 1.67 and in E_H is 1.33, both are lower than O_P; however, results still suggest a high perception of ease for the overall Hand-as-a-Prop. O_P is perceived as the easiest translation, but Hand-as-a-Prop translation is also perceived as easy to perform.

The perceived translation ease is slightly higher for S_H than E_H, but the SD is also higher. It somehow suggests that both are perceived as easy to perform, but such a perception might be affected by the sense of touch, highly depending on each person.

5.5 Fatigue

Results showed a significant effect on fatigue (F(2,11) = 9.525, p = 0.001, η2 = 0.462), depending on the type of haptic feedback. Post-hoc comparisons using Bonferroni corrections showed significant differences between S_H and O_P or E_H.

In every condition, fatigue of the right hand and arm as reported in questionnaires was similar (see Fig. 8). In the S_H condition, fatigue of the left hand (M = −2, SD = 1.2) and left arm (M = −2, SD = 1.04) were higher than in E_H (Left Hand: M = −3, SD = 0, Left Arm: M = −2.92, SD = 0.29, p < 0.006) and O_P (Left Hand: M = −3, SD = 0, Left Arm: M = −3, SD = 0, p < 0.002) as reported in the questionnaires. In E_H and O_P, the left hand and arm fatigue was less than in the right, but not significantly. In all cases, the perceived fatigue is low, suggesting that none of the techniques causes significant fatigue.

5.6 User preference

User preference was collected in a rank order questionnaire (Fig. 9). The preferred condition was O_P (chosen in the first place 75% of the times). S_H was the second preferred condition and was preferred in the first place 25% of the times. One participant who chose S_H as the preferred one expressed that “grabbing the object was easier than in the other techniques because I knew where my left hand was”, this suggests the benefits that proprioception might bring to this task. The E_H was the least preferred technique, it was never selected as the preferred one, and it was the most selected in the third position (66%). It seems that this technique does not feel real enough and at the same time is not easy to perform. Some participants expressed that it was difficult to move the external agent hand toward the target position, especially with the cylinder: “I had to tilt the hand in order to reach the target”, “the cylinder was more complex to place on the target and this made the exercise more difficult”.

Fig. 9
figure 9

Haptic feedback condition preference for the translation task (1 = most preferred, 3 = least preferred)

Full size image

6 Discussion

The goal of this work was to determine the feasibility and impact of using hands (either the own hands or an external person hands) to provide haptic feedback for objects with different shapes in Virtual Reality in the context of a manipulation task. To this aim, we conducted a user study evaluating user experience and performance using qualitative and quantitative methods. Next, we discuss the main implications of using External or Self Hands-as-a-Prop, and compare them with traditional Object Props, in terms of their representation and translation capabilities.

6.1 Representation of different object shapes

Through the exploration task, we investigated if different shapes can be represented by hand props, specifically, a cube, a cylinder, and a flat surface, similar shapes were used in shape-changing devices (McClelland et al. 2017).

User questionnaires indicate, as expected, that a real object prop is perceived as the best match and the most realistic representation. The opinions of users toward using an external “human agent” hand (E_H) or using their own hand (S_H) to represent such shapes are lower but similar. The perception of realism and representation goodness for E_H and S_H was neutral-to-positive.

The cube and flat objects are easily represented by the human hand, whereas the cylinder is considered harder. The cylinder was represented only by one finger (see Fig. 2 Real Hand) highlighting some of the limitations of the Hand-as-a-Prop technique that should be considered when designing the map between hand gestures and objects. Even when the three gestures were rated as easy to perform (under the S_H condition), goodness and realism results suggest that shaping capabilities of the human hand are more suitable to mimic a cube and a flat surface than a cylinder. In general, O_P provided the best user experience while S_H and E_H were assessed as neutral-positive.

6.2 Manipulation performance and user experience

One of the objectives of this research was to evaluate if it was possible to provide haptic feedback using only the hand (no props). We evaluated only manipulation performance in a translation task as recommended by Bergström et al. (2021). The results showed that manipulation times are similar between Self Hand-as-a-Prop and Object Props but significantly higher in the External Hand-as-a-Prop condition. This means that moving someone else’s hand takes more time than moving an object prop or your own hand. Specifically, in the S_H case, the non-dominant hand might help to coordinate the translation movement and make it faster. When analyzing the Full TCT, the extra time to detect the non-dominant hand gesture might have influenced the user’s performance, meaning that by optimizing gesture detection, S_H would provide even better results than E_H.

Realism and ease of the sensation of holding and translating the virtual object showed significantly worse results in both Hand-as-a-Prop than in Object Props. However, as in the exploration tasks, results suggest that Hand-as-a-Prop is an easy to perform technique for translating objects, and realism was perceived as neutral-to-positive. All the techniques were assessed with a low fatigue perception. Object Prop provides the best user experience but Hand-as-a-Prop results are still positive.

When we compare Self Hand-as-a-Prop with External Hand-as-a-Prop, the former technique does not require any external device or person. We might think that receiving external assistance leads to a better perception of the experience compared to using our own bodies. However, results indicate that there is no difference in terms of realism and ease during the translation task between Self and External Hand-as-a-Prop, suggesting that perception of realism might not be affected by the sense of touch from your “own hand” but just from using a hand, any hand.

Lastly, in terms of preference, O_P is the preferred technique. S_H was preferred over E_H, which was the least preferred technique. This preference of S_H over E_H might be interpreted as a partial consequence of proprioception “self hand was easier than in the other techniques because I felt where my left hand was”.

7 Summary

Overall, Hand-as-a-Prop (self or external) showed acceptable performance and it provided a neutral-to-positive User Experience in comparison with Object Props. Hand-as-a-Prop is a valid alternative to traditional static props for a virtual reality manipulation task. We conclude that Self Hand-as-a-Prop presents better results in terms of user experience and performance than External Hand-as-a-Prop, and it has the main benefit that it does not need external actuation (from a robot or a person).

Our initial expectations were that External Hand-as-a-Prop might provide a more realistic experience than Self Hand-as-a-Prop because users will automatically find the object in the natural expected position, as in regular Object Props. We also expected Self Hand-as-a-Prop to present a better performance in terms of completion time since users could move both hands harmonically. However, only the second assumption was proven true, the realism of the External Hand-as-a-Prop was not better than in Self. The External technique presents limitations in the translation tasks and the representation of different shapes compared to the Self Hand-as-a-Prop.

The perceived realism is affected because the props are human hands and the participants noticed it since the hand does not represent the virtual shape perfectly, it is an approximation. Many participants complained about the cylinder representation under this condition and one of them also mentioned that “the only thing that makes me feel not immersed is the temperature of the hand, I expected something colder”. On the other hand, the translation can be problematic because moving someone else's hands might imply a kind of non-verbal negotiation in order to agree in the direction, orientation, and speed of the movement. Actually, one of the participants explained that “External hand neither feels like a real object nor like something that has its own movement, it is confusing”.

7.1 Limitations and future work

In this work, we evaluated the feasibility of using Hand-as-a-Prop within a manipulation task. In the study users were seated and only able to rotate the torso, head and arms, but walking was not considered. Further research should be done in this situation. Regarding the variety of the objects involved in the study, we need to incorporate other objects with different sizes and shapes. The proposed technique is not suitable for tasks that require high haptic resolution such as surgery training. It is neither viable for simulating large objects like walls or doors in architecture design.

Results for Self Hand-as-a-Prop technique might be biased by the hand gesture detection algorithm. Ideally, the gesture detection algorithm should be trained with the hands of each participant in order to improve its accuracy. Furthermore, the findings related to Self and External Hand-as-a-Prop might be biased by the Oculus built-in hand tracking algorithm since when hands overlap sometimes the recognition fails.

In the translation task under the External Hand-as-a-Prop technique, the external agent might bias the results by exerting more or less resistance to the user’s movements or even unconsciously helping in the translation. Actually, one of the participants mentioned that “It felt less real, sometimes I had to tilt the external agent’s hand in order to make the system detect that the object landed on the target”. We understand that being the external agent requires certain training and expertise in order to avoid any kind of interference in the user’s intended movements. Furthermore, another limitation of the External Hand-as-a-Prop technique is that it requires a display where the external agent checks the position of the virtual object that has to prop.

Lastly, in the Self Hand-as-a-Prop technique, the single hand manipulation is an intrinsic limitation of the technique, i.e., two objects cannot be manipulated independently by each hand.

Future work includes exploring how other object shapes can be represented by the hand, and whether visual dominance dominates over tactile shape as proposed by Rock and Victor (1964) and implementing and evaluating techniques with real-world complex tasks.

8 Conclusion

We designed and implemented a technique for providing haptic feedback while manipulating objects in virtual reality environments. The technique Hand-as-a-Prop employs a hand (the user or that from another person) as a proxy for the virtual object. We performed a user study to compare Hand-as-a-Prop with the traditional Object Props approach under a two-task experiment. The study showed that using Object Props provides the best user experience and performance within a manipulation task. Hand-as-a-Prop took more time to complete the tasks, with the exception of the manipulation time of using your own hand, which was similar to Object Props. The technique proved to be a feasible alternative which was rated with a neutral-to-positive user experience. Self Hand-as-a-prop provided better results than External Hand-as-a-Prop with the benefit of not requiring the presence of any external agent. In particular, users preferred the Self Hand-as-a-Prop technique over the External Hand-as-a-Prop for a translation task.

We hope that the results presented here would guide VR application developers in using the user’s own hand as a representation for the virtual objects when some haptic feedback is required. When the VR user is accompanied by a colleague or helper, the External Hand-as-a-prop method will be useful if there are no prop objects available for example in casual gaming in the street or inspection of 3D models in remote locations. The Self Hand-as-a-prop method will be useful when no instructor or helper is available to provide haptics, for example in applications that require grasping and placing objects, i.e., self-training of machinery operators or instrumentation placement; or playing casual games alone without having to break immersion when searching for physical objects.