Keywords

1 Introduction

An interface for manipulating avatars is an essential component of virtual reality (VR). In science fiction works, VR is often described as a system in which users can subjectively move freely while their physical body is lying on a bed. However, present VR systems reflect the movement of the physical body to the avatar using devices such as position tracking controllers.

In VR, the movement of the physical body causes various problems. For example, the risk of collision and injury; the requirement for a large space and equipment; and the problem that people with limited mobility cannot use VR. The development of a VR system that does not require physical movement would lead to the solution of such problems of safety, cost, and accessibility.

A brain-computer interface (BCI) can be used to implement a movement-free VR system. However, BCI has technical difficulties, such as the requirement for expensive devices, and invasive means (e.g., anesthesia) are required to block motor commands.

By contrast, a simpler method may exist. Even if the user’s body is physically fixed, by altering user’s kinesthesia (sense of body movement), the user may be able to feel moving as if not fixed. Fortunately, our kinesthesia can be modulated relatively easily. For example, the sense of motion can be generated even if the body is not actually moving if vibration stimulus is applied to the tendon [6]. This phenomenon is called kinesthetic illusion.

In this study, we propose a system that detects the force exerted by a user’s body fixed on a rigid frame to control an avatar, and presents kinesthetic illusion using tendon vibration. In this paper, We implement a system limited to one degree-of-freedom motion as a proof of concept, and evaluate its operability and effects on body ownership and agency.

2 Related Work

When vibration is applied to the tendon of a muscle, kinesthetic illusion, in which the vibrated muscle is stretched, is generated [6]. This phenomenon is considered to be caused by the activation of the muscle spindle by vibration [6], and it has been reported that the nerve fires at a frequency that agrees with the vibration frequency up to a certain vibration frequency [4, 15].

Research has also been conducted on the presentation of complex kinesthetic illusions using this phenomenon. Albert et al. [1] converted recorded nerve firing patterns to vibration and presented kinesthetic illusions that reproduced the movement in the recording. Additionally, Thyrion et al. [16] successfully presented three-dimensional kinesthetic illusion by predicting nerve firing the movement trajectory and converting it into vibration.

This phenomenon has also been applied to VR and human-computer interaction. For example, Hagimori et al. [7] combined tendon vibration with a visual stimulus using a head-mounted display (HMD) to make small physical movements perceived as large movements in VR. Barsotti et al. [3] proposed a system that combined kinesthetic feedback using tendon vibration with a BCI based on motor imagery.

Among the above techniques, in the BCI-based approach, the user does not need to move physically at all. However, in those BCI systems, the user has to perform motor imagery, rather than trying to move physically. This may cause a sense of unnaturalness in VR applications.

Other methods exist to modulate kinesthesia in addition to tendon vibration. Okabe et al. [13] reported that the illusion of finger movement was generated by presenting the flow field of tactile sensation according to the shearing force of the fingertip. Similarly, Heo et al. [8] generated the illusion of bending an object using vibrotactile stimuli according to the force applied to the object.

The modulation of a kinesthetic sensation can also result from visual stimuli alone. Pseudo-haptic feedback proposed by Lecuyer et al. [10] is a technique to present a sense of force only using visual stimuli. However, it was reported that a force-sensing stationary device, similar to our study, was perceived like moving in their experiment [10].

As a method similar to our study, Mochizuki et al. [12] proposed a VR interface that does not require physical movement by measuring the torque exerted by joints of the physically fixed user. However, their current system used visual stimulus only, and techniques that directly alter kinesthesia were not tested.

3 System

3.1 Operating Principle

The operation of the proposed system is shown in Fig. 1. In this system, the user’s body (arms, legs, etc.) is fixed to a rigid frame. When the user tries to move the body, a force sensor attached at a fixing position detects the force exerted by the user (Fig. 1a). In response to the detected force, the computer simulates the user’s intended motion and renders a virtual arm on the HMD (Fig. 1b). Simultaneously, a vibration stimulus is applied to the user’s tendon in response to the simulated motion to present kinesthetic illusion (Fig. 1c). Thus, vision and kinesthesia are presented as if the user is moving the body, despite the user’s body being fixed.

Fig. 1.
figure 1

Operation of the system (a) user tries to move and exerts a force. (b) Computer simulates movement and controls the HMD and vibrators. (c) Vibration induces illusory movement.

Full size image

3.2 Implementation

As a proof of concept of the proposed method, we implemented a system limited to one degree-of-freedom motion. Although this technique may be applicable to the motion of various joints, we chose the extension and flexion of the forearm following the experiment of Roll et al. [14].

System Configuration. The system detects force using a load cell attached to the wrist fixing component, and connected to the PC through the front-end circuit. The game engine Unity (Unity Technologies) was used for the simulation and rendering, and visual stimuli were presented using an HMD Vive Pro (HTC Corporation) via an external graphics processing unit. The vibration waveform was generated using a waveform generator circuit, amplified using an audio amplifier (MUSE M50), and presented using two vibrators (Vp210, Acouve Laboratory). The vibrators were attached near the right elbow using an elastic fabric supporter, to vibrate distal tendons of the biceps brachii (BB) and triceps brachii (TB) muscles. To make the initial position of the virtual arm coincide with the physical arm, a position tracking device (Vive Tracker, HTC Corporation) was attached to the frame. Additionally, to enhance the ownership of the arm, the actual movement of the user’s fingers was captured and reproduced in the display using Leap Motion (Ultraleap).

Algorithms. The angular velocity of the virtual elbow joint was proportional to the force applied to the load cell; that is, when the force applied to the load cell was \(F\,[\mathrm {N}]\) (assuming \(F=0\) at system startup), the commanded value of the angular velocity was \(\omega _\mathrm {command} = 50 F \,[\mathrm {deg/s}]\). The angle of the elbow joint \(\theta \,[\mathrm {deg}]\) was obtained by integrating the angular velocity, but was limited to \(-45\) to 45\(^\circ \). For the angle and angular velocity, the extension direction was positive. Vibration waveforms were generated by frequency modulating a sine wave between 0 and 100 Hz. Although the amplitude changed according to the frequency because of the frequency response of the system, and kinesthetic illusion is likely to be diminished in lower frequencies, a simple linear mapping was used for the sake of simplicity. This algorithm is based on the knowledge that nerve firing corresponds to the vibration frequency [4, 15] and its applications [1, 16]. The vibration frequencies \(f_\mathrm {BB}\,[\mathrm {Hz}]\) (for BB) and \(f_\mathrm {TB}\,[\mathrm {Hz}]\) (for TB) were

$$\begin{aligned} f_\mathrm {BB}= & {} \left\{ \begin{array}{ll} 0 &{} (4 \omega< 0) \\ 4 \omega &{} (0 \le 4 \omega < 100) \\ 100 &{} (\mathrm {otherwise}) \end{array}\right. \end{aligned}$$
(1)
$$\begin{aligned} f_\mathrm {TB}= & {} \left\{ \begin{array}{ll} 0 &{} (-4 \omega< 0) \\ -4 \omega &{} (0 \le -4 \omega < 100) \\ 100 &{} (\mathrm {otherwise}) \end{array}\right. \end{aligned}$$
(2)

where \(\omega \,[\mathrm {deg/s}]\) was the angular velocity of the virtual elbow joint (\(\omega = 0\) when the angle \(\theta \) reaches positive or negative limit). However, the coefficients used in the above algorithms were set empirically and not determined theoretically.

4 Experiment

To evaluate the operability, body ownership, and agency of the avatar using the proposed system, we conducted an experiment in which the participants used the system under the following three conditions.

  • Tendon All elements of the proposed method were incorporated.

  • None Only visual stimulus was used, without vibration stimulus.

  • Tactile Only cutaneous cues were presented using high-frequency vibration that was unlikely to cause kinesthetic illusion. Based on the method of Bark et al. [2], an amplitude-modulated sine wave of 250 Hz was used for the vibration waveform, and the amplitudes of the waveform input to the BB and TB vibrators were \(A_\mathrm {BB} = 0.1 \times 10^{\theta / 45}\) and \(A_\mathrm {TB} = 0.1 \times 10^{-\theta / 45}\) (from 0 to 1), where \(\theta \,[\mathrm {deg}]\) is the virtual elbow angle.

4.1 Methods

Initially, 13 laboratory members that specialize in VR and/or haptics participated in the experiment, but because malfunctions of Leap Motion occurred during experiments involving three of the participants, their data were excluded from subsequent analyses. Finally, data from 10 participants (22 to 25 years old, average 23.6 years old, all right-handed, one female, nine male) were analyzed.

First, the acceleration amplitude of the BB vibrator attached to the participant’s arm driven with a 100 Hz sine wave was adjusted to approximately \(130\,\mathrm {m/s^2}\) (which was determined to stably evoke kinesthetic illusion), using an accelerometer (LIS331HH, STMicroelectronics). We also presented 100 Hz sine wave stimuli and confirmed orally that the kinesthetic illusion occured in both the extension and flexion directions. When the illusion was not sufficiently obtained, the position of the vibrators was re-adjusted until the illusion occurred.

The participants then performed a task to evaluate the operability of the system based on Fitts’ law [5, 11] for each condition. The task consisted of 50 trials. In each trial, the participants controlled the virtual forearm and kept the angle aligned with the target for 1 s. The center position of the target was randomly generated from \(-30^\circ \) to \(30^\circ \) and the width was randomly generated from \(5^\circ \) to \(15^\circ \). White noise was presented using the built-in headphones of the HMD during task execution.

After the task for each single condition was complete, the participants answered the questions shown in Table 1 using the seven-point Likert scale (−3: totally disagree, +3: totally agree), to evaluate body ownership and agency. This questionnaire was a modified version of the questionnaire used in the study on rubber hand illusion by Kalkert et al. [9] and consisted of four categories: Ownership, Ownership Control, Agency, and Agency Control. The questions were translated into Japanese.

Table 1. Questionnaire for evaluating ownership and agency (based on Kalckert et al. [9])
Full size table

To cancel the order effect, the order of the conditions was counterbalanced as much as possible. However, due to the aforementioned malfunctions, among \(3!=6\) permutations, orders of conditions None-Tendon-Tactile and Tactile-Tendon-None was used only once, and the other orders were used twice.

4.2 Results

Operability Evaluation Using Fitts’ Law. In the task results for each participant and each condition, the equation of the modified Fitts’ law MT = \(a+b\log _2(A/W+1)\) [11] was fitted, where \(MT\,[\mathrm {s}]\) is the time required for the trial, \(A\,[\mathrm {deg}]\) is the difference between the angle at the start of the trial and the target angle, and \(W\,[\mathrm {deg}]\) is the width of the target. Additionally, the index of performance IP was calculated [11]. Figure 2 compares the average IP for all participants under each condition. The repeated measures analysis of variance showed no significant differences between the conditions (\(p=.898\)). As a result of multiple comparisons using the Bonferroni correction, no significant differences were found (\(p=1.000\) for all pairs). Additionally, the correlation coefficient r of the fitting ranged from 0.139 to 0.851 with an average of 0.532.

Questionnaires. In the same manner as [9], the answers to questions belonging to the same category were averaged, and four scores (Ownership, Ownership Control, Agency, and Agency Control) were calculated (Fig. 3) and further analyses were done using these scores. The Wilcoxon signed-rank test between Ownership and Ownership Control, Agency and Agency Control was performed in each condition and significant differences in both Ownership-Ownership Control and Agency-Agency Control were found in all conditions (\(p<.05\)). Also, the Friedman test was performed between conditions for each score, but there was no significant difference (\(p=.393\) for Ownership, \(p=.087\) for Ownership Control, \(p=.607\) for Agency, \(p=.098\) for Agency Control).

Fig. 2.
figure 2

Comparison of the average IP by condition (error bars indicate the standard deviation)

Full size image
Fig. 3.
figure 3

Comparison of the questionnaire scores by condition

Full size image

4.3 Discussion

Operability. There was no significant difference in IP between the conditions, and as shown in Fig. 2, the average IP for the Tactile and Tendon conditions were almost the same or slightly lower than that of the None condition. Hence, we consider that operability was not improved by presenting kinesthetic illusion in the proposed system.

Body Ownership and Agency. For the results of the questionnaire, there was no significant difference in scores between the conditions, possibly due to large variability between participants. However, in Fig. 3, ownership tended to increase in the Tendon condition in comparison with other conditions. Therefore, while we cannot conclude that tendon vibration was effective in the movement-free VR interface, it may contribute to the generation of ownership.

Additionally, in Fig. 3, a small decrease in Agency and an increase in Agency Control in the Tendon condition were observed. In fact, some participants’ comments suggested a lack of agency, such as “a feeling of being moved by others” for the Tendon condition. We consider that the loss of agency was because movement simulation and the method for the vibration presentation were imperfect, and the kinesthetic illusion was different from the intended motion of the user.

5 Conclusion

In this paper, we proposed and implemented a VR system that requires no physical body movement by detecting the force exerted by the user’s fixed body and presenting kinesthetic illusion. As a result of the experiment, an improvement in operability caused by kinesthetic illusion was not confirmed. Additionally, although not statistically verified, the results suggest that the kinesthetic illusion may lead to an improvement of ownership and decrease of agency. As future work, a more precise verification of usefulness and improvement of the presentation method are required.