Keywords

1 Introduction

Virtual reality environments are a growing application area, with fully immersive haptic-audio-visual environments (HAVE) being employed in diverse areas such as training and gaming. However, it is possible that adding more realism can create a decrease in performance [1] or even cause impairment [2]. With the increasing popularity of HAVE environments, it becomes crucial to study how adding different modalities of sensory input can impact the user’s ability to carry out tasks. In an extreme case, the impact on the user could be severe enough that they should be discouraged from activities such as operating heavy equipment or even driving.

In this study, the impact of tactile feedback in HAVE environments will be examined. The HAVE environment consists of a fully immersive head-tracking Oculus Rift to provide the visual. The sound is played through headphones and tactile feedback is supplied by a haptic chair from TADs Inc.

Section 2 gives the background. A brief description of the experimental apparatus is described in Sect. 3 and Sect. 4 describes the VR environment that the subjects are immersed in. Section 5 will present an analysis of the pilot study, while the last chapter will present conclusions and future research.

2 Background

Although the technology exists to increase realism in virtual reality (VR) environments, it can come with a cost. In [1], it was demonstrated that, in taking a 2D visual display to a 3D display, a demonstrable decrease performance in a command and control application was measured. In attempting to create more realism using immersive VR, nausea and other side effects can be introduced [2].

There are many ways to leverage the somatosensory system to create a more immersive experience for the user in VR environments, including vibrations, motion, and force feedback, which help the user to suspend disbelief and increase the realism of the experience. These devices come in different forms, and use a variety of touch-based sensations to create physical sensations and feedback from the VR world for the user.

The TactaVest [3] is a wearable system that uses tactors (tactile transducers) to provide the user with specialized information to enhance VR interactions. Events taking place in the scene can be translated to the user through vibrations in respective locations on the body. Wearable systems are an effective way to provide full body haptic experiences to the user, but these may also be problematic when developing systems for multiple users. For example, sizing devices for different body types, housing the cables, providing enough power to drive the system, and ensuring the transducers are properly positioned on the body. Wearable haptic devices in VR may also provide mobility to the user, but can also pose safety issues for users who are not able to see the actual environment, which can lead to tripping on or disconnecting cables. The Surround haptics system [4], developed by Disney Labs is a chair-based tactile display that uses tactors to create haptic sensations. This system uses a 4\(\,\times \,\)3 array of tactors to create a variety of haptic sensations for the user, which are used to increase the connection a user has to a game or VR environment by leveraging the different illusions that can be created through haptic vibrations. Another approach to improving the VR experience is to create motion for the user. While motion actuators can be used to provide the user with the sense that they are moving along with the action taking place in VR, this can often be expensive, and also lead to motion sickness in some users. While this is a problem inherent in the use of VR displays [5], motion in the seat may increase nausea in users, despite the potential benefits it may have for the user experience. An alternative approach to using actual motion actuators to move a chair was developed for the HapSeat system [6]. This approach leverages the kinesthetic and vestibular mechanisms of the somatosensory system to provide the user with a sense of movement, without actually moving the seat. This is accomplished by applying force feedback to the arms and head, which may effectively trick the user into believing that they are moving simply through the displacement of their limbs. By placing force feedback devices on the arms and head rest of the chair, the force feedback sensations can cause the user to feel like they are moving, but with potentially less risk of causing motion sickness. Since most of the haptic systems described rely on the use of mechanical vibrations and force feedback sensations that are not linked to real world experiences, it is necessary to create mappings of vibrations onto the events or experiences they are linking to in the virtual world. While researchers are discovering and developing new sensations and libraries of different haptic sensations [7], it is challenging to create more naturalistic experiences that the user can feel and intuitively map onto the events they are intended to be feeling.

In a different approach to providing real time tactile feedback to users, sound originating from the virtual world or scene can be used as tactile vibrations on the body. The Emoti-Chair was first developed in 2008 to provide deaf and hard of hearing people with access to sounds from movies and music entertainment [8]. Studies into this type of sensation showed that sounds could be communicated to the body as a natural form of input, and effectively identified and linked to the source of the sound. In addition, information pertaining to the emotional expression and quality of the sound could naturally be detected by users, who in most cases were deaf and may never have heard sounds before, but could still interpret the emotional content [9].

Tactile acoustic devices (TADs) are based on the model of the human cochlea, which theorizes that by presenting sounds on the skin in a similar manner that the human cochlea uses to process sounds one can effectively turn the skin into a hearing organ [10]. Because of the ability to directly communicate sound to the body, it is possible to create tactile sensations that can lead to an immediate recognition of the mappings between the vibrations and the sounds.

Given the often sound-rich environments associated with VR, the TAD system may be a natural extension to the existing haptic devices and systems gaining traction within the VR domain. While this is a novel application for the TAD technology, its ability to integrate into the content of most VR scenarios increases its potential to quickly provide relevant and important tactile information to the user, and to possibly increase and enhance the overall experience of virtual world interactions.

Extending this technology to the VR world could potentially offer more interesting and naturalistic sensations to users, which may also work in combination with other haptic and tactile devices to provide a richer experience for the user.

Fig. 1.
figure 1

Setup of roller coaster experiment

Full size image

3 Experimental Method

In this pilot study, subjects are asked to sit in a fully immersive audio-visual environment and go through a realistic computer simulated roller coaster simulation (see Fig. 1). All subjects will experience an identical ride. The 3D visual component is enabled through head-tracking VR Oculus Rift goggles, and the stereo sound component is supplied via stereo headphones. All tests will be run with the subjects sitting in a system developed by Tactile Audio Displays Inc. (TADsInc), which combines an X-Rocker Pro 4.1 gaming seat [11, 12] that has been augmented with an 8-channel TAD (Tactile Acoustic Device) and a low frequency bass transducer (as well as speakers) built into the seat. This is a chair that has 12 tactile sound transducers situated along the back and seat, with half situated to the left or right of the spine (see Fig. 2). The tactile sensations are generated using sound to simulate the vibrations that sounds generate and transmit through the seat of a roller coaster. This is based on providing tactile sound to the body as an approach to creating a sense of enhanced realism for the user, who can experience the ’feeling’ of the sound of the roller coaster on their body.

Fig. 2.
figure 2

TAD X-Rocker chair

Full size image

Participants in the study are first asked to sit down in the Haptic chair while one of two scenarios (tactile sensation on, tactile sensation off) are chosen for them at random. The participant will then watch a black screen on the Oculus Rift display for 3 min 14 s. After the time has elapsed, the participant is asked to move into the viewing space of the NaturalPoint OptiTrack motion capture system [15], and is told to stand as still as possible for one minute as their motion sway is recorded (see Fig. 3). This consists of recording the relative position of 3 infra-red markers placed on the back of the subject’s head. After the recording is complete, the participant is told to sit back down in the chair, and is fitted with headphones. The participant then experiences the roller coaster simulator with either the haptic sensation on or off. After watching the simulation for 3 min 14 s, the participant is told to move into the OptiTrack viewable area. The participant is once again told to stand as still as possible for one minute as their sway is recorded. After the experiment is complete, the participant is told to complete two questionnaires; the IPQ Presence Questionnaire [16] and the Motion Sickness Assessment Questionnaire [18]. The IPQ Presence Questionnaire is used to measure the sense of presence experienced in a VR environment, and the Motion Sickness Questionnaire asks the participant to qualitatively rate their impairment from the study. After the completion of the experiment and questionnaires, the participant is told to come back for a second session where they will perform the experiment with haptics off, if they did the first experiment with haptics on, or vice versa. The second session follows the same procedure as the first, including the completion of the same questionnaires.

Fig. 3.
figure 3

Capturing motion sway using optitrack

Full size image

4 VR Environment

To study the effects of VR impairment, a roller coaster simulator was chosen as the VR content for this study. This simulator virtually places the participant in the front cart of a roller coaster while the cart moves at a predetermined rate along the tracks. The ride repeats itself continuously until the program is closed automatically by a timer. An image of the roller coaster environment in shown in Fig. 4. This VR environment was created in the Unity game engine platform using assets from the asset store [13, 14], and the Oculus Rift Unity plugin. As the environment was created in a game engine, the user has an immersive, spherical view of the environment. Additionally, due to the head-tracking hardware of the Oculus Rift, the user can move their head around in the virtual environment. The simulator is run on an Intel Core i5 CPU and a Nvidia GeForce GTX 980Ti GPU. A high performance GPU is being used to minimize lag effects which may be present when using the Oculus Rift. The audio from the VR environment is fed into the TAD haptic chair’s audio mixer, using a 3.5 mm audio cable, to create tactile sensations. Headphones are also connected to the chair to allow the participant to hear the sounds coming from the roller coaster. The Oculus Rift is connected to the computer using an HDMI cable and is integrated with the VR environment using the aforementioned Oculus Rift Unity plugin.

Fig. 4.
figure 4

Roller coaster VR environment through Oculus Rift

Full size image

5 Pilot Study

Six participants were recruited for this pilot experiment. Of the six participants, 5 are male and 1 is female, and all University of Waterloo students. All participants are regular computer users, however, their experience with VR environments varied greatly. This study was approved by the Office of Research Ethics at the University of Waterloo who allowed the recruitment of participants using both posters and university administered e-mails.

The quantitative data collected in this study consists of the time-series recording of the anterior motion sway of participants. This is done capturing the position of three infra-red markers, placed on the back of the participants’ heads, using the NaturalPoint OptiTrack system. To qualitatively assess motion sickness, data is collected from two Likert-type questionnaires. One being the Igroup Presence Questionnaire (IPQ) to assess perceived presence, and the other being the Motion Sickness Assessment Questionnaire (MSAQ), which allows participants to express how sick they felt. Linear regression is conducted for time series position data (x, y, z). Mean Square Error (MSE) is calculated for the residual values from the line of best fit corresponding to the subject’s motion sway in the anterior direction, i.e. \(\frac{1}{n}\sum _{i=1}^n(z-z_0)^2\). Only movement in the z-direction is used for the MSE calculations; however, due to the alignment of the participant with the OptiTrack system, anterior motion is not always solely in the z-direction. As a result, these calculations do not fully encapsulate total motion sway. A sample graph of a participant’s motion with the line of best fit is shown in Fig. 5. Figure 6 depicts the average MSE for six participants. There is a difference between the Haptics ON condition and the Haptics OFF for all markers. The Haptics ON to the Haptics OFF average ratios (Eq. 1) are 0.76, 0.86, 0.84 for Marker 1, Marker 2 and Marker 3 respectively. As can be seen in Fig. 6, the MSE for Marker 1 in the Black 1 experiment is quite large at \(1.5{\times }10^{-4}\text {mm}^2\). This is due to an outlier data point. Instead, using only 5 participants’ data in the MSE calculations, the MSE for Marker 1 drops to \({1.03{\times }10^{-4}}\text {mm}^2\), which is more in line with the other results.

$$\begin{aligned} MSE_{ratio}=\frac{1}{n}\sum _{i=1}^n\frac{MSE_{i_{HapticsON}}}{MSE_{i_{HapticsOFF}}} \end{aligned}$$
(1)
Fig. 5.
figure 5

Pilot study participant’s anterior sway and corresponding line of best fit

Full size image
Fig. 6.
figure 6

Average Mean Square Error (MSE) for each condition: position is recorded for infra-red markers attached to the Oculus Rift VR goggles (Color figure online)

Full size image
Fig. 7.
figure 7

Average Motion Sickness and average Presence: data collected from Igroup Presence Questionnaire (IPQ) presence questionnaire and Motion Sickness Assessment Questionnaire (MSAQ) (Color figure online)

Full size image

The questionnaire data supports the hypothesis that there is no difference between the Haptics ON and Haptics OFF experimental conditions although differences are quantitatively confirmed by the position data analysis. As can be seen in Fig. 7, there an insignificant difference in Motion Sickness between the Haptics OFF and Haptics ON. The results use only the raw data from the questionnaires.

5.1 Sample Size

The sample size of this study is estimated using an t-statistic iterative method [17]. An initial z-test is conducted to estimate the initial degrees of freedom by replacing the t-statistic power and confidence level with z-statistic coefficients. Multiple t-tests were computed to refine the degrees of freedoms at each iteration for the smallest difference between the obtained value and desired true value to be detected. For the second iteration, the degrees of freedom of the first iteration is used to look up the value of the level of confidence and statistical power. Since each participant volunteers for two sessions, the number of participants required to achieve statistical significance is approximately 37 using Eq. 2. The results of the calculations are shown in Table 1.

$$\begin{aligned} n=\frac{2(t_\alpha +t_\beta )^2s^2}{d^2} \end{aligned}$$
(2)

Here, \(\alpha \) is the confidence level (\(95\,\%\) for the study), \(\beta \) is the statistical power (\(80\,\%\) for the study, s is the standard deviation, d is the smallest difference detected between the obtained and desired true values, \(t_\alpha \) is two-sided t-statistic for \(\alpha \)-value of 0.05, and \(t_\beta \) is on-sided t-statistic for \(\beta \)-value of 0.2.

Table 1. Sample size calculation based on a pilot study involving 6 participants
Full size table

6 Conclusion and Future Work

This pilot test is an initial study into the impact of tactile sensations on HAVEs. The preliminary results indicate that the addition of tactile sensations in this application, in order to increase realism, does appear to decrease impairment when the tactile haptic sensations are on. This is despite the fact that the subjects’ perceived presence and motion sickness, captured through questionnaires, seem to indicate no difference between the haptics on and off scenarios. This indicates that a more detailed study should be conducted. The calculations indicate that over four times more participants need to be recruited. This will be conducted in future research.