The way of water: exploring the role of interaction elements in usability challenges with in-car VR experience

Abstract

With advancements in autonomous driving technology, the variety of activities that can be performed in a vehicle has increased. This improves the possibility of watching virtual reality (VR) content on a head-mounted display (HMD). However, unlike VR used in stationary environments, in-car VR can lead to discomfort and motion sickness due to the vehicle movements. Additionally, the obstruction of the outside view during driving may cause user anxiety. In this study, we investigated, for the first time, the effect of dynamic road environments, such as turns, stops, and speed bumps, on the in-car VR experience. Based on our findings, we included situational awareness (SA) cues in the in-car VR content to help users perceive their surroundings and improve the user experience. We conducted a user study with thirty participants to validate the impact of these cues. Consequently, we discovered that the Dynamics cue, which provides SA information while maintaining the context of the VR content, improves user immersion and trust while easing VR motion sickness.

1 Introduction

Fig. 1

Concept images of the proposed in-car VR: a baseline with no interaction, b content with map and voice guidance, c content with adaptive dynamics elements, d game content with controller, and e content with additional audiovisual stimulation

With the development of autonomous driving systems, long-distance commuters can engage in activities other than driving, as they do not have to participate in dynamic driving tasks after boarding a vehicle (SAE I 2021). Steck et al. (2018) found that the experience of self-driving vehicles is perceived to be similar to the time spent on public transportation. They provided empirical evidence that the value of travel time savings for commuter experiences could be reduced by 31%. In such situations, we can consider using the head-mounted display (HMD) as a leisure activity in an autonomous vehicle. Hock et al. (2017) and Paredes et al. (2018) argued that using virtual reality (VR) in the car could improve the travel experience. Hock et al. (2017) developed an in-car VR system to investigate simulator sickness, impressions, and enjoyment as a function of vehicle motion. Participants reported that conditions with vehicle motion were more exciting because the visuals and motion were consistent, and it was confirmed that simulator sickness was reduced. However, forward visibility may be obstructed when using HMD inside vehicles; this makes it difficult to predict the direction of the vehicle’s movement, leading to discomfort and motion sickness. Some participants in the studies by Hock et al. (2017) and Paredes et al. (2018) reported that they felt most uncomfortable when they suddenly applied the brakes while the vehicle was moving. These researchers argued that simulation sickness increases when the content is designed without visual cues. Another participant inquired about the real driving situation in the vehicle; in response, the researchers acknowledged that this phenomenon could affect appreciation. Some participants frequently raised the HMD during the experiment to check the situation outside the vehicle. Currently, the test environment for in-car VR studies has certain limitations. The road environment used for the test consisted of a traffic-free parking lot, a straight section, and a curved section. However, typical driving situations include events that cause dynamic changes in vehicle motion, such as turns, stops, and speed bumps, which are different from VR experiences in a room or stationary space. According to Li et al. (2021), whether the road is paved and traffic conditions, such as stopping and driving style, affect motion sickness. Goedicke et al. (2018) argued that road texture and surface conditions should also be considered when implementing a virtual environment. Therefore, it is important to inform users about external conditions while they are experiencing VR content in a moving vehicle. Pöhlmann et al. (2022) further recommended that providing visual cues that simulate motion related to driving can reduce motion sickness and improve the user’s immersion.

In this study, we divided our research into two phases. Firstly, we investigated the specific road environments, such as turns, stops, and speed bumps, which affect the usability of in-car VR. In the main study, based on the road environments investigated in the pilot study, we explored the effect of delivering visual cues about external driving situations on the usability of VR. Our goal was to analyze the relationships between driving events, interaction factors, and user experiences. We also aimed to propose interaction design considerations for content development that can be used with HMD in vehicle driving situations (Fig. 1). This is the first study to examine the effects of the proposed interaction factors on in-car VR usability in dynamic driving scenarios, providing valuable insights for future research in the field.

2 Related works

2.1 In-car virtual reality

VR technology has the potential to enhance the flexibility, efficiency, and productivity of workspaces (Grubert et al. 2018). When combined with autonomous driving technology, it can create innovative experiences in a mobile environment. This has led to an increasing amount of research investigating the use of VR technology in vehicles. Hock et al. (2017) announced the in-car VR system, which allows users to enjoy VR in a car by mapping vehicle movements and visual information. They confirmed simulator sickness, presence, and enjoyment under two types of car-driving conditions. In this study, participants wore a VR HMD in the passenger seat, and driving routes with curves and 360-degree turns were used in the experiment. Under the driving conditions, where the movement was caused by driving, scores for enjoyment and presence were higher than under parking conditions, and nausea was lower. Kodama et al. (2017) developed a virtual reality entertainment prototype system composed of an HMD, one-person electric car, and automatic driving algorithm. In their indoor user study, all users reported a positive experience. McGill et al. (2017) confirmed how the degree of correspondence between visual display and vehicle motion affects motion sickness. Participants in the passenger seat used a VR HMD to enjoy video content under six conditions. The experiment was conducted on a quiet one-way road without traffic lights. A new baseline in terms of VR motion sickness in cars has been established, and it has been argued that the use of VR HMD in vehicles makes sense. In addition, Goedicke et al. (2018) observed user experience by developing a VR driving simulation platform to study vehicle interaction or interaction on a real road. They set up a driving route, including four curves, and conducted semi-structured interviews related to the user experience with the participants. Paredes et al. (2018) argued that in-car VR could provide a calming experience for passengers. First, they compared the static and dynamic content of two vehicle movements. Then, they observed which content participants endured longer when the car was in motion. Driving routes were used in the experiment, including residential, highway, and rural mountain roads. As a result, viewing dynamic content in a moving car calmed people. Li et al. (2021) conducted a rear-seat VR field study on a 4-km-long highway and explored the effects of the range of head movement on the usability of an in-car VR, such as comfort, motion sickness, and engagement. While some studies have explored the benefits of VR in cars on the road, their test routes had limitations in terms of components. We analyzed driving routes with events, such as turns, stops, speed bumps, and rough roads, which cause motion changes in dynamic vehicles.

2.2 Wizard of Oz in driving simulations

Various driving simulation platforms have been suggested to analyze the interaction and behavior of autonomous vehicles and passengers. Himmels et al. (2022) emphasized the significance of high-quality simulators that closely replicate actual driving situations to improve participant immersion. Some platforms imitate driving on real roads to provide a more realistic experience. Yeo et al. (2020) performed an analysis of six autonomous driving simulators, ranging from indoor simulations to real vehicle-based simulations, based on visual and motion fidelity. They discovered that higher visual and motion fidelity resulted in a greater sensation of being present in the simulation environment. In the case of real vehicle-based autonomous driving simulations, a Wizard of Oz (WoZ) driver is required owing to issues related to safety and ethical regulations (Yeo et al. 2019). Baltodano et al. (2015) and Wang et al. (2017) developed low-cost and portable WoZ driving systems to simulate autonomous driving by using partitions to conceal the wizard driver. Goedicke et al. (2018) employed a similar system, where passengers wearing VR HMDs felt as if they were steering the car while a driving wizard was actually in control. They also claimed that the VR-OOM system could be used to observe behavior in specific scenarios with autonomous vehicles. McGill et al. (2022) compared various features of the existing in-car extended reality (XR) platforms and developed PassengXR, a low-cost and open-source toolkit. This toolkit allows individuals to experience gaming, productivity, and collaboration within vehicles. In our study, to simulate a more immersive autonomous driving scenario, we utilized two wizard drivers. One wizard driver drove the vehicle from the back seat before the participants put on the VR HMD to make them believe it was an actual autonomous vehicle. The other driver controlled the vehicle from the driver’s seat after the participants had put on the VR HMD.

2.3 Situation awareness

As wearing an HMD obstructs the view of the passenger, the information they can obtain is limited. In a study by Goedicke et al. (2018), some participants lifted the HMD to check the situation while riding in a vehicle because they were curious about the external situation. Moreover, some of the pilot study participants wanted the camera screen filming the external situation to be displayed at the bottom. Therefore, the degree of wanting to know external information and the delivery method may be one element to consider when designing in-car VR content. The original concept of situational awareness (SA) is related to pilots and aircraft (Fracker 1988; Vidulich et al. 1994). According to Endsley (2000), SA is simply defined as “knowing what is going on around you.” This is what you need to know when making decisions to achieve your goals. For example, if your goal is to reach your destination, you need to know where your car and obstacles are, when to turn, and how the road conditions are changing to avoid potential hazards. According to Rolnick and Lubow (1991) and Diels and Bos (2016), the ability to predict what might happen in the future is related to motion sickness in cars. When using a VR HMD in a vehicle, simulator sickness increases when there are no cues about the car and the orientation of the player (Hock et al. 2017). Manipulating the VR view to induce anticipatory actions helps prevent motion sickness (McGill et al. 2017). However, providing inappropriate SA cues tailored to the content may not have a significant impact on motion sickness prevention. Yusof et al. (2020) attempted to reduce motion sickness by increasing SA while watching videos about self-driving cars. They used a vibrotactile display to provide haptic feedback during a turning event. Compared to the baseline condition, SA increased, but motion sickness did not decrease. They found that although participants knew the car was turning, it did not decrease motion sickness sufficiently because they did not know when or why it would turn. Conversely, Cho and Kim (2022) observed motion sickness when experiencing a virtual office in the car, either showing the road environment transparently in the background of the content or suggesting a method to utilize particle flow. The motion sickness was found to be significantly less than that in the baseline conditions. Pöhlmann et al. (2022) also concluded that synchronized visual motion in VR headsets can effectively reduce motion sickness for vehicle passengers. Furthermore, as observed by Fereydooni et al. (2022), SA cues that are not optimized for the content environment may effectively convey SA information, but they can impose a significant cognitive load when receiving information. This can potentially diminish the sense of immersion in the content. Therefore, a method is required to provide SA cues that prevent motion sickness without compromising the sense of immersion in the in-car VR content. According to Endsley (2000), SA can be conveyed through visual, auditory, tactile, olfactory, and gustatory cues, which can be explicit or subtle. In the case of in-car VR content, diegetic cues can be employed to attract users’ attention to the immersion of the content and cues. Rothe and Hußmann (2018) demonstrated the effectiveness of using signals such as sound and light to engage users in immersive virtual environments such as cinematic VR. Other studies have shown that such narrative cues not only facilitate intuitive and seamless interaction (Beck and Rothe 2021; Dickinson et al. 2021) but also contribute to a rewarding immersive experience (Marre et al. 2021) and enhance the efficiency of the learning experience (Dickinson et al. 2021). Therefore, through diegetic cues, SA cues can maintain user immersion in the content along with SA information in in-car VR.

Hence, in our study, while experiencing in-vehicle VR content, we first evaluated the driving situations that necessitate SA cues and observed the changes in user experiences, including presence, motion sickness, trust, and discomfort, based on the type of SA cues. We provided SA cues in two ways: the first involved providing SA information directly through familiar navigation systems, such as maps and voice guidance; and the second utilized diegetic cues, incorporating objects within the virtual environment linked to the content, to convey SA alongside driving information. Further, we investigated the impact of SA cues optimized for in-car VR content.

2.4 Presence and immersion with in-car VR

In a virtual environment, the sense of presence is the perception of being physically present (Witmer and Singer 1998). The attributes of the virtual environment and the user’s interpretation considerably influence this sense of presence (Skarbez et al. 2017). In immersion, a VR system’s technical and design features enhance this feeling of presence. The accuracy of sensory feedback, the level of interactivity, and the system’s ability to accurately simulate or depict reality are critical immersion attributes. With increased immersion, the perceived sense of presence within the virtual environment increases (Witmer and Singer 1998). To reduce anxiety by providing external information, we investigated how to enable users to fully immerse themselves in the in-car VR content experience. Weech et al. (2019) stated that higher levels of presence are associated with lower levels of cybersickness. The first proposal is interaction using physical activity. The use of interactive technology in a virtual environment plays an important role in feeling present (Slater and Usoh 1994). McMahan (2013) argued that interfaces that use kinesthetic senses, such as hearing and sight, bring a high level of presence. Han and Kim (2017) compared gaze-based hand interaction and gaze only to increase immersion. Although the difference was small, interaction using both gaze and hand provided better satisfaction and immersion. They also reported that there were no significant differences between the two methods in terms of motion sickness. Harth et al. (2018) also argued that interaction using tracking technology, controllers, virtual hands, and avatars in a virtual environment could lead to immersion. Interaction with content enhances immersion, but passengers inside vehicles have limited space for movement. Therefore, it is necessary to have an interaction technology that can enhance presence and immersion even in constrained spaces. Kari and Holz (2023) achieved full-range 3D input even in narrow spaces by using smartphone position and orientation along with touch input. Wilson et al. (2023) assessed the usability of interaction techniques, such as Linear Gain, Gaze-Supported Remote Hand, and AlphaCursor, which are suitable for confined spaces such as transport seating. They found that Gaze-Based or AlphaCursor interactions were more effective in terms of inducing presence and trust. Therefore, to induce fun and deeper presence in participants through interaction, the AlphaCursor-based interaction condition was selected as one experimental condition. The following is a method that uses distraction. According to Malloy and Milling (2010), VR distraction can be used to solve various pain problems, such as pain unpleasantness, and reduce the time spent thinking about pain. Bos (2015) used mental distractions to alleviate motion sickness and found that reported sickness decreased when thoughts about sickness were intentionally interrupted. In addition, when experiencing a virtual environment in a restricted space with noise, a distraction from the cause of discomfort improves comfort and is more effective when enjoying the virtual environment (Lewis et al. 2016). However, when enjoying in-car VR content, discomfort, such as motion sickness due to vehicle shaking, may be induced. We added a distraction condition using the diegetic cue, expecting the sense of immersion to be improved by dispersing the attention focused on this discomfort.

3 Implementation

Fig. 2

On-road in-car VR platform

Our platform is designed to provide an immersive VR experience for passengers in autonomous vehicles (Fig. 2). It has been successfully used in both pilot and main experiments. Users can enjoy immersive content that reflects driving events while sitting in the passenger seat of an autonomous vehicle. When users put on a VR headset, they enter a virtual environment where they can explore the underwater world in a submarine. We chose the underwater environment as the main theme of the content for two reasons. First, underwater scenes can alleviate motion sickness and anxiety by providing a comfortable and calm environment. According to Paredes et al. (2018), underwater scenes provide an immersive experience and a calming and mindful experience for passengers of vehicles moving with unique visual elements, such as underwater creatures, tranquil scenery, and vivid colors. Second, novel and attractive content can arouse the curiosity of participants and help them immerse themselves. Lewis et al. (2016) reduced discomfort by distracting participants using underwater scenes with unique visuals and exploration opportunities, which can be effective in diverting participants from potential sources of discomfort. The submarine in the virtual environment, which corresponds to the autonomous vehicle, receives driving information such as the position, speed, acceleration, and rotation of the autonomous vehicle and moves in the same manner within the content. After the participant wore the VR headset, the experimenter in the driver’s seat steered the vehicle along a predetermined path. Participants received a visual experience through VR and an auditory experience through noise-canceling earphones. Noise cancelation was used to minimize the noise generated by the hardware system inside and outside of the vehicle. The condition involving the game provided the experience of interacting with objects in the virtual environment through a controller.

3.1 Hardware components

3.1.1 VR headset

We used the Oculus Quest 2, which supports Unity3D. It supports a resolution of 1832 × 1920 pixels in both eyes and a refresh rate of up to 120 Hz. After connecting to the PC using the Oculus Link function, the content was played by running Unity. Depending on the user, a face band with a nose cover was used to prevent the outside view from being seen, as the nose piece was not fully attached to the VR headset.

3.1.2 Earphone

To suppress the noise generated while driving and maximize the visual experience of the content, Apple AirPods Pro 1, which can suppress noise, were used. Sound was reproduced by transmitting the audio source to the earphone through Oculus Link.

3.1.3 Computer

A laptop was used for controlling the VR content system. ASUS’s MSI GE66 (Intel 9-11980 HK, NVIDIA GeForce RTX 3080) was connected to the VR HMD via Oculus Link and was used to communicate global positioning system (GPS) and inertial measurement unit (IMU) sensor data received through Arduino with Unity. Samsung’s NT930SBE was used as the wizard laptop for the WoZ, which uses a different laptop because it can be run in an Ubuntu environment. By connecting Polysync DriveKit with a wizard laptop and a Kia Soul EV, an environment was created where driving could be performed via a joystick outside the driver seat, and WoZ was implemented to prevent the participant from driving. Participants were shown that they could manipulate autonomous vehicles and in-car VR content through a tablet located at the center of the vehicle.

3.1.4 Sensors

GPS-RTK Dead Reckoning Breakout-ZED-F9R (Qwiic) from SparkFun was used as the GPS and IMU in our system. The module onboard the Arduino was attached and used in the center of the vehicle. The GPS data were transmitted to the module from an antenna installed outside the vehicle, and the IMU data were directly measured using the module. Arduino transmitted GPS and IMU data as serial messages through user datagram protocol (UDP) communication using a laptop computer and shared the data by reading it in Unity.

3.2 Software components

The following elements make up the Unity scene of XR contents: objects that collide with vehicles and users (car and XR Player), driving conditions and objects that control events (condition controller, interaction manager, driving event, and sensor receiver), and an object (sea map) that represents the topography and visual effects of the virtual sea.

3.2.1 Car

This is a game object that corresponds to an autonomous vehicle and moves according to the trajectory and rotation of the vehicle while carrying the user. Based on the data measured by the sensor module, the motion trajectories of the autonomous vehicle and the car object are synchronized.

3.2.2 XR player

This game object provided by the XR Interaction toolkit faces the user wearing a VR headset. It moves while riding on a car object. The movement of the user (translational and rotational movements) in the autonomous vehicle and the XR Player in the car object are synchronized in real time.

3.2.3 Condition controller

As a script component of the car object, it controls which condition to use in each execution. Elements such as game objects and events are created and used according to the selected condition by adjusting the Boolean value of the condition to be used before our system is run.

Interaction Manager: The different conditions of our system are controlled by each manager, and the interaction manager is an abstract class of all managers. If the car object collides with the driving event, the event-start function is executed. Each manager overrides the interaction manager when implementing specific functions.

3.2.4 Driving event

As a game object corresponding to all driving events that can occur while the autonomous vehicle is moving along a predetermined path, it is placed at a predetermined location in a virtual environment. The sets of driving events are right turn, left turn, bump, stop, and rough road. The name of the game object was recorded differently so that the manager script could recognize each event. When the collider of the car object and the driving event object are executed on the trigger entry, the event start of the interaction manager is executed.

3.2.5 Sensor receiver

This is the game object responsible for serial communication with the sensor module. The object processes GPS and IMU data according to the scale of the virtual environment and determines the extent to which the car object needs to rotate and move.

3.2.6 Sea map

This game object contains visual elements in a virtual environment. It includes groups of objects, such as cliffs, caves, and corals, that show various terrain changes.

4 Study 1: pilot

When using a VR HMD, especially in a moving vehicle, people may experience motion sickness due to the discrepancy between bodily sensations and visual information. Additionally, anxiety may be induced when wearing an HMD to experience a virtual environment in a car because the front view is blocked and the sense of immersion may change because of a shaking vehicle. To address these issues, it is crucial to provide effective SA cues. However, a comprehensive assessment of the specific road situations requiring these cues during driving scenarios is currently lacking. In the pilot study, we chose a driving route that induces various motion changes in the vehicle. Our goal was to explore when and why the user experience is interrupted when participants enjoy VR content in the car. Through interviews, we confirmed the necessity of delivering SA cues. Based on the results of the pilot study, we will consider a method that does not interfere with the enjoyment of in-car VR content, select it as a variable, and evaluate its effect in the main study.

4.1 Participants

Eighteen participants (7 females and 11 males) participated in this experiment (18–32 years; M = 27.8, SD = 6.3). They were recruited through a regional job-posting platform. Three of them had never experienced a VR device before. All participants except one had a driver’s license, and 11 of these participants (1–18 years; M = 3.75, SD = 5.41) had actual driving experience.

4.2 Study design

Fig. 3

Driving route: A is a turn, B is a bump, C is a stop point, and D is an uneven road

A within-subject design was used. To assess the need for SA cues during the in-car VR content experience, we chose a route that replicates diverse driving scenarios. The driving route used for the experiment was approximately 1.4 km long and included 4 right turns, 4 left turns, 4 stops, 9 speed bumps, and 4 rough roads, and it took approximately 6 min to drive (Fig. 3). The in-car VR content consisted of a virtual submarine synchronized with the vehicle’s movement. Participants freely explored underwater scenarios, detached from driving conditions and without SA cues. We collected responses to the survey questions and conducted interviews to answer them.

4.3 Procedure

Participants who arrived at the experimental site signed the informed consent form after being informed of the purpose of the experiment and the data collected during the experiment. Participants completed the simulator sickness questionnaire (SSQ) upon entering the vehicle and received instructions for using the VR HMD to explore underwater content. The task was to press the button on the controller if the participant viewing VR content felt uncomfortable while riding in the vehicle. At this point, any form of discomfort was allowed including motion sickness, anxiety, and vigorous vehicle movement. Participants were informed that they could stop the experiment at any time if they experienced severe motion sickness. Participants sat in the passenger seat of the vehicle. Three experimenters sat in the driver’s and back seats of the vehicle (Fig. 2). Participants adjusted the passenger seat to a comfortable angle and adjusted the focus and head strap of the HMD. After they donned the HMD, the vehicle began to move. For approximately 6 min, the VR content was played while the vehicle drove along a predetermined path on the campus. In the meantime, participants worked to press the button on the controller when they felt uncomfortable. After the VR content was finished and the vehicle stopped, participants filled out the SSQ again. Subsequently, they watched a recorded video of the driving situation with the experimenter and conducted an interview. The video showed the moment when participants felt uncomfortable and pressed the button. At that moment, the experimenter paused the video and asked a question. The interviews lasted approximately 20 min, and the study lasted approximately 40 min.

4.4 Measurements

Through the pre-survey, we investigated whether they had car sickness, VR experience, VR sickness, anxiety about self-driving cars, a driver’s license, or driving experience. We used the SSQ for simulator sickness measurements before and after the experiment. After the survey was completed, a semi-structured interview was conducted with the following content: the reason you pressed the button at that moment, how to improve the discomfort, questions about the situation outside the vehicle, and memorable moments while enjoying the VR.

4.5 Results

Fig. 4

Results of the simulator sickness questionnaire (SSQ) Total score and subscores (Nausea, Oculomotor, Disorientation) (left); and the interview results indicating reported discomfort (right) (*p < 0.05, **p < 0.01; Error bars denote standard error (SE))

Table 1 Statistical analysis of simulator sickness questionnaire (SSQ) scores before and after

We evaluated participants’ simulator sickness levels, including nausea (N), oculomotor-related discomfort (O), and disorientation (D), using the SSQ and subscores. The results of the Paired Samples T-test on the SSQ evaluations before and after the experiment revealed significant findings, as illustrated in Fig. 4 on the left and Table 1. A significant increase was observed in the Total score. Subscores for Nausea and Disorientation also showed significant increases for motion sickness. However, the Oculomotor subscore did not yield significant differences and exhibited a small effect. Through semi-structured interviews, we analyzed the impact of driving situations, such as rotation, stops, and speed bumps, on participants’ experiences and the reasons for pressing the button. Among the 18 participants, as shown in Fig. 4 on the right, 9 reported discomfort during rotations, 12 during stops, and 10 during speed bumps. For rotations, they experienced discomfort due to abrupt rotations, uncertainty about safe rotations, and an inability to discern external situations. They mentioned, “I suddenly rotated, causing discomfort and motion sickness symptoms.” (P7, P12, P16, P18), “During rotation, virtual obstacles (rocks, seaweed) were present, causing uncertainty about when to rotate and maintaining safety.” (P3, P11), and “I felt anxious because I didn’t know the exact reasons for rotation and the external circumstances.” (P9). Regarding stops, even though virtual objects associated with stopping were not visible, participants often experienced sudden halts leading to motion sickness due to the mismatch between visual and physical sensations. They stated, “There were no obstacles on the screen, but sudden halts made me uncomfortable, causing motion sickness.” (P1, P6, P7, P18) and “I experienced jolts and motion sickness during sudden stops in unpredictable situations in the content.” (P8, P10, P12, P17). Lastly, concerning speed bumps, participants noted significant discomfort and heightened motion sickness due to the lack of visual cues for speed bumps within the virtual environment, causing unpredictable and severe motion sickness. They reported, “Although there’s nothing on the ground and the underwater surface moves like sliding, sudden jolts make me uncomfortable.” (P1, P3, P6, P11, P17), “When crossing speed bumps, there’s no way to directly confirm where they’ll appear, leading to sudden jolts and motion sickness.” (P2, P7, P10), and “Speed bumps cause excessive shaking, making it hard to concentrate and leading to intense dizziness from the mismatched movements.” (P12, P16, P18).

4.6 Implications for the main study

In our pilot experiment, participants reported discomfort and indicated a need for SA cues during various driving events that caused dynamic body movement changes. We chose driving events such as turns, stops, rough roads, and speed bumps as points where SA cues should be provided. Effective SA cues in in-car VR experiences can reduce motion sickness and anxiety, as reported by Pöhlmann et al. (2022) and Fereydooni et al. (2022). Diegetic cues, demonstrated by Rothe and Hußmann (2018) and Dickinson et al. (2021), have been found to enhance content immersion and can be used as SA cues. In the main study, we plan to address SA cues by incorporating SA information into user-friendly navigation systems, such as maps and voice guidance (C2: Navigation). We will also use objects in the virtual environment that are linked to content to provide SA alongside driving information as diegetic cues (C3: Dynamics). To improve immersion and reduce discomfort when experiencing VR content in a car, we developed game conditions that require controller-based interaction (C4: Game), such as AlphaCursor (Wilson et al. 2023). Additionally, we suggest a condition (C5: Distraction) that utilizes diegetic cues to enhance immersion in situations where SA cues are necessary, thereby helping to shift the focus away from uncomfortable driving scenarios. In the main study, we will observe how users’ experiences, such as presence, motion sickness, trust, and anxiety, are affected by the delivery of SA cues in five different conditions, including a baseline condition where no information is provided.

5 Study 2: main

Through a pilot study, we confirmed that user experience might be hindered when enjoying in-car VR content on a driving path that causes vehicle motion changes. Therefore, in the main study, we aimed to select a method that can reduce user experience disruptions based on the insights obtained through the pilot study interview and previous studies and test its effectiveness.

5.1 Participants

Thirty participants (11 females and 19 males) participated in this experiment (18–52 years; M = 29.3, SD = 8.9). They were recruited through a regional job-posting platform. Eight participants had never experienced VR devices before. Except for four, all participants had a driver’s license, with 18 individuals having actual driving experience (1–30 years; M = 5.08, SD = 8.16).

5.2 Study design

Fig. 5

a Driving route. A is a turn, B is a bump, C is a stop point, and D is an uneven road. b Using Unity, we created a virtual underwater world with rocks, a sandy floor, and other elements that match the roads seen in the aerial photographs of the area

The study was conducted with a within-subject design. The driving route used in the main experiment was inside the campus, and it was approximately 1.1 km long, including 4 rotations, 2 stops, 4 speed bumps, and 3 rough roads. It took approximately 4 min to drive (Fig. 5 (a)). Our research questions were as follows:

• RQ 1: Which interactive elements affect the user experience improvement, such as motion sickness and presence, when experiencing VR content in a car?

• RQ 2: When providing SA of external driving conditions, is there any difference in the user experience of in-car VR appreciation depending on the provision method?

In the experiment, five types of in-car VR content were used. The five conditions were counterbalanced by using a Balanced Latin Square.

5.2.1 C1 (Baseline)

This is a baseline condition without elements, such as road and driving situation information and interaction. The player explores a virtual sea underwater, which is implemented according to the real-world road configuration at the same speed as that of the vehicle (Fig. 6).

Fig. 6

Condition 1 (Baseline): a participants sitting in the passenger seat, and b the underwater world without any additional effects

5.2.2 C2 (Map)

Some participants in the study by Goedicke et al. (2018) removed their HMD during the experiment because they were curious about the external situation. From this, we selected conditions that provided information about the external situation. The first enables the recognition of the exterior information of the vehicle, such as roads and driving situations, and provides information in a way that is familiar to the user. A map indicating a driving route and the current location was visually provided on a display in a virtual environment, and voice guidance was used simultaneously (Fig. 7). We expected that providing SA in this way would increase immersion by reducing the anxiety of users.

Fig. 7

Condition 2 (Map): a map UI (user interface), and b a display is displayed at the bottom of the submarine, with maps and traffic signs aligned to their current position

5.2.3 C3 (Dynamics)

A condition was added to provide SA differently, increasing immersion while decreasing motion sickness by using gaze flow and context. For instance, when rotating, a group of fish moves ahead of the player in the direction of the movement of the vehicle. At the stopping point, rocks or sunken ships are broken, and the field of view is blocked for a while. At the location of the speed bumps, underwater fountains rise upward. A rough road section is represented as a relatively high terrain with corals (Fig. 8). Abstract information about the environment of the vehicle is provided to the user by movement changes of the elements that compose the VR content while adapting to the movement of the vehicle. As in the examples above, they are expressed by internal elements of the VR content (our content theme is underwater exploration, so we used objects that make up the seabed, such as fish and rocks), without using exact terms or signs such as “speed bumps.”

Fig. 8

Condition 3 (Dynamics): a Ship rocking scene when the vehicle comes to a stop; b View of a coral reef area in a rough road section; c Scene where a group of fish moves ahead of the player in the direction of the movement of the vehicle when turning right; d Scene of underwater fountains rising upward when the car goes through speed bumps

5.2.4 C4 (Game)

We added this condition to increase the sense of immersion and presence in the VR content. We used physical interactions to allow participants to focus on the content itself, rather than informing them of external driving situations. Twenty-four square gems were placed in the virtual environment. Participants used a controller to aim at the jewelry and then pressed a button to collect it (Fig. 9).

Fig. 9

Condition 4 (Game): a After aiming the red light at the jewel and pressing the button on the controller, it collects the jewel, and the game dashboard UI is displayed on the bottom display; b a participant using the controller

5.2.5 C5 (Distraction)

This condition diverts attention from the inconvenience of driving, such as vehicle shaking, and focuses on the VR. Additional visual and auditory elements have been created to enrich the experience. Particle effects, such as light pouring, are added, and fish of different types and sizes are created. Background music other than the existing underwater sound effects was also added (Fig. 10).

Fig. 10

Condition 5 (Distraction): When a driving event occurs, information about the external situation is not provided. Fish, light effects, and sound effects occur to distract users

5.3 Procedure

Participants who arrived at the experimental site signed the consent form after receiving an explanation of the purpose of the experiment and the data collected in the experiment. Participants were told that there were five types of content exploring the sea after boarding a vehicle and wearing a VR HMD and that there were questions to answer in the questionnaire after the experience. Participants were informed that they could stop the experiment at any time if they suffered from severe motion sickness. Participants were also informed that the vehicles used in the experiment were driven autonomously. This explains why only predefined routes were operated autonomously, and experimenters sat in the driver’s seat to avoid safety issues. The participants got into the passenger seat of the vehicle. Three experimenters, including a wizard driver, were in the driver’s and back seats of the vehicle (Fig. 2). Before wearing the HMD, a test drive was conducted to confirm that the vehicle was operating autonomously. The wizard driver in the back seat drove the vehicle to the starting position of the content path for approximately 30 s. During the test drive, the vehicle moved forward and turned right until it came to a smooth stop without the experimenter holding the steering wheel in the driver’s seat. After the wizard driver finished driving, participants wore the headset by adjusting the focus and head strap of the HMD. After wearing the HMD, the experimenter in the driver’s seat assumed control of the vehicle and began to drive it. As a result, participants believed that the vehicle was moving autonomously. For approximately 4 min, the VR content was played while the vehicle drove along a predetermined path on the campus. After the VR content ended and the vehicle stopped, the participants completed a survey. This process was repeated five times. After the five runs and completion of the questionnaire, a semi-structured interview was conducted. The study, illustrated in Fig. 11, lasted approximately 90 min.

Fig. 11

Experimental procedure in the main study

5.4 Measurements

Four indicators were selected for the user experience evaluation. The SSQ, which displays 16 symptoms on a 4-point scale, was used to measure simulator sickness (Kennedy et al. 1993). The Igroup Presence Questionnaire (IPQ) was used to measure the “feeling of being there” or being present. The IPQ comprises 14 items, including general presence, spatial presence, involvement, and realism (Schubert et al. 2001). A Trust in Automation (TiA) questionnaire was used to measure system reliability (Körber 2018). We measured the reliability of the system because we used the WoZ method to make participants think that the vehicle drives autonomously. It is divided into 6 sub-elements and consists of 19 items. To measure anxiety, the Flight Anxiety Modality (FAM) questionnaire is used (Van Gerwen et al. 1999). Because it is a questionnaire based on flight situations, it is conducted after fine-tuning some of the questions for self-driving cars with reference to the study of Koilias et al. (2019). It consists of 11 somatic modality scales and 7 cognitive modality scales of anxiety. To confirm the interaction design of the five conditions, the Situation Awareness Rating Technique (SART), which measures situational awareness (Taylor 2017), and the NASA Task Load Index (NASA-TLX), which measures workload, were also used (Hart and Staveland 1988).

6 Results

Fig. 12

Total scores of Situation Awareness (SART) and Presence (IPQ) for each condition (*p < 0.05, **p < 0.01; Error bars denote standard error (SE))

Table 2 Statistical analysis of SART and IPQ scores across different conditions

6.1 Differences in situation awareness

In this section, we report the comparison of the SART scores among conditions used when participants experienced the in-car VR content (Fig. 12, left). We identified differences in participants’ SA based on the type of cue used in the in-car VR content. Specifically, by comparing the conditions with SA cues (C2 (Map), C3 (Dynamics)) to C1 (Baseline), we discerned the influence of SA cues on participants’ SA. Additionally, the impact of the conditions providing other cues (C4 (Game), C5 (Distraction)) on participants’ SA was further validated.

The SART values between C1 (baseline) and the other four types of cues were compared. After validating the data distribution’s normality through kurtosis and skewness, a repeated measures ANOVA (RM ANOVA) analysis was conducted according to the within factor: cue types (C1 (Baseline) to C5 (Distraction)). Subsequently, upon identifying the violation of Mauchly’s Test of Sphericity, we employed the Greenhouse–Geisser correction to adjust the degrees of freedom. Furthermore, to ascertain changes in SA due to the cues used in the in-car VR content, conditions with cues were pairwise compared to the C1 (Baseline) using the Bonferroni’s post hoc test (Table 2).

From the RM ANOVA analysis of SART values based on the cues used in the in-car VR content, a significant difference in participants’ SA among the conditions was observed (F(3.017, 87.482) = 2.866, p = 0.041, η² = 0.09). This result indicates that the level of SA can vary depending on the cues provided when participants experience in-car VR content. Additionally, the pairwise comparison with C1 (Baseline) revealed a trend of increased SA in the two conditions providing SA cues (C2 (Map), C3 (Dynamics)). Specifically, C2 (Map) revealed a tendency for increased SA with a small effect size. C3 (Dynamics) significantly increased SA with a medium effect size. By contrast, the other two conditions (C4 (Game), C5 (Distraction)) did not produce any changes in SA. This indicates that C3 (Dynamics) condition, designed to enhance SA for in-car VR content users, effectively provided information about the driving situation. Additionally, cues providing irrelevant information about the surroundings do not impact the users’ SA.

This indicates that C3 (Dynamics) condition, which was designed to enhance SA for in-car VR content users, effectively provided information about the driving situation. Additionally, cues that provide irrelevant information about the surroundings do not impact the users’ SA.

6.2 Differences in presence

We compared IPQ values across different in-car VR exposure conditions to identify differences in participant presence (Fig. 12, right). We found that participants experienced different levels of presence depending on the cues in the VR content. We observed changes in presence caused by cues that were specifically designed to enhance presence. Similar to the previous section, to compare IPQ values between C1 (Baseline) and the other four conditions, we ran an RM ANOVA and a Bonferroni’s post hoc test (Table 2).

Results from the RM ANOVA analysis on the IPQ values, based on the cues provided in the VR content, indicated a statistically significant difference (F(4, 116) = 4.971, p < 0.001, η² = 0.146). This suggests that the sense of presence perceived by participants varied depending on the provided cue. Pairwise comparisons of C2 (Map) with both C3 (Dynamics) and C4 (Game) revealed statistically significant differences, featuring a medium effect size in IPQ values. Meanwhile, the comparison between C2 (Map) and C5 (Distraction) showed a trend towards an increase with a smaller effect size. These results indicated that the cues used in C3 (Dynamics) and C4 (Game) can be more effective in enhancing presence in in-car VR content compared to other cues, whereas the cues in C2 (Map) did not significantly affect presence.

6.3 Results on user experience in-car VR content

This section presents findings from interviews and questionnaires evaluating the user experience with in-car VR content across different cue types. We assessed the rankings of conditions based on user experiences, including factors such as immersion, trust, and sickness. We also analyzed interview feedback to evaluate the usability and appropriateness of five conditions for implementation in in-car VR content. Furthermore, the questionnaires provided additional insights into user experiences, enhancing our comprehension of their interactions with the in-car VR content.

Subsequent analyses compared differences in each measure based on the type of cue applied to the in-car VR content. We checked the data for normality by observing kurtosis and skewness. We then proceeded with an RM ANOVA based on the within factor of cue type. When Mauchly’s Test of Sphericity was violated, we employed the Greenhouse–Geisser correction for the degrees of freedom. If there were differences in measures due to the factor, we conducted pairwise comparisons using Bonferroni’s post hoc test.

6.3.1 Interview results

Fig. 13

Ranking scores of immersion, trust, and sickness for each condition. The number of respondents is 30, 24, and 26, respectively (*p < 0.05, **p < 0.01, ***p < 0.001; Error bars denote standard error (SE))

Table 3 Statistical analysis of immersion and trust ranking scores across different conditions

Immersion ranking

RM ANOVA analysis showed a significant difference in immersion ranking scores among participants based on the cue type factor (F(4, 116) = 18.750, p < 0.001, η² = 0.393). Post hoc comparisons revealed that the ranking scores for C3 (Dynamics), C4 (Game), and C5 (Distraction) were notably higher. Specifically, the ranking score of C3 (Dynamics), C4 (Game), and C5 (Distraction) were approximately 1.87, 2.4, and 1.57. This suggests that while C3 (Dynamics), C4 (Game), and C5 (Distraction) can enhance user immersion in in-car VR content, C2 (Map) does not significantly alter content immersion (Fig. 13 (left), Table 3).

Immersion interview

Our interview revealed variations in the perception of immersiveness among the 29 participants, depending on the type of cue factor involved in the in-car VR content. The most immersive in-car VR experience was reported under the C4 (Game). Twenty-four individuals reported heightened immersiveness due to game-related cues. P4 stated, “I had other thoughts about when the other conditions would end, but I didn’t think about this condition and became fully immersed in it.”, and P24 noted, “I became immersed in the idea that I had to collect all the gems, as the number of collected gems was displayed.”. Additionally, 17 participants reported that the C5 (Distraction) was also immersive, while only three individuals experienced decreased immersiveness in C5 (Distraction). For instance, P12 mentioned, “I loved that there was so much to see, and it felt super real, especially when the fish came close to my face!” and P20 said, “I got the feeling of experiencing virtual reality properly because there were many fish appearances and various music came out.”. The C3 (Dynamics) also contributed to a more immersive in-car VR experience for 14 participants. P12 commented, “That was a really cool experience because of the fun effects like falling rocks and the way they showed the bumpy road even when we were driving under the sea.” and P26 added, “The VR made it look like the car was bouncing on the rough parts of the road and speed bumps, and it was pretty fun to anticipate the shaking. It also helped me get more into the content.” By contrast, only one and four participants reported C1 (Baseline) and C2 (Map) to be immersive, respectively. Participants 6 and 25 reported that these conditions negatively impacted the immersive experience. P6 stated, “The voice guidance woke me up from the immersion in the content and disturbed me.” and P25 mentioned, “I ended up focusing more on the navigation than the ocean, so it didn’t really help with experiencing the underwater world.”.

Trust ranking

Similarly, trust ranking scores varied based on the cue type factor (F(4, 92) = 9.439, p < 0.001, η² = 0.291). Specifically, the trust ranking scores under the C2 (Map) were statistically significantly higher than those under C1 (Baseline), C3 (Dynamics), and C5 (Distraction). The score for the C4 (Game) was significantly higher than for C1 (Baseline). Although the comparison between C2 (Map) and C4 (Game) did not reveal statistical significance, the trend suggests that C2 (Map) tended to provoke higher trust than C4 (Game), with a medium effect size. This pattern of results indicated that users had the highest trust in the cues provided by C2 (Map), followed by C4 (Game). However, the difference between these two was not statistically significant (Fig. 13 (middle), Table 3).

Trust interview

Our study investigated whether participants felt any unease or perceived a lack of safety while engaging with the VR content. Through the interviews, we gathered feedback related to trust from 20 participants based on different cue types. The C2 (Map) generated feelings of stability and safety for many participants. Specifically, 15 participants reported a sense of reliability with this cue. However, two participants expressed feelings of unease due to the navigation information. For instance, P11 noted, “The car moved just like the navigation said, which made me trust it more.” whereas P30 added, “They gave a heads-up about what was coming on the road, so I felt mentally prepared.”. The C3 (Dynamics) was generally perceived as offering a safe experience, with nine participants citing its integration of real-world traffic situations into the content as fostering feelings of safety. Only two participants provided negative feedback regarding this cue’s stability. P5 mentioned, “Following the direction the fish was showing made me feel secure, and it was a cool experience.” while P25 said, “Visually showing the rough roads and speed bumps helped cut down the anxious moments and got me more into the content.”. Feedback on the C1 (Baseline), C4 (Game), and C5 (Distraction) was less favorable, receiving only 3, 4, and 2 positive comments, respectively. Other participants either had no noteworthy experiences regarding stability or provided negative feedback. For example, regarding the C1 (Baseline), P11 expressed, “It freaks me out when the car jolts. Not knowing when a speed bump is coming up made me feel dizzy and anxious.”

Sickness ranking

Contrarily, there was no statistical difference observed in the sickness ranking scores (F(2.945, 73.635) = 0.459, p = 0.708, η² = 0.018), suggesting that participants did not experience significant differences in sickness regardless of cue type compared to the C1 (Baseline) (Fig. 13 (right), Table 3).

Sickness interview

To investigate how VR content influences users’ experiences of sickness, such as motion sickness or dizziness, we interviewed participants based on the type of cue they encountered. Feedback on sickness was recorded from 24 participants across different cue types. Participants who encountered the C4 (Game) had mixed experiences regarding sickness. Nine participants reported that focusing on their mission deterred them from recognizing sickness-inducing elements, resulting in a positive experience. However, five participants felt fatigue and dizziness due to the need to concentrate on finding gems in the game. For example, P29 mentioned, “I was so focused on finding the gem that I didn’t notice anything else, which reduced the motion sickness.” while P3 expressed, “Focusing on a distant gem, with the car’s movement shaking the screen, felt like it was messing with my head.”. The C3 (Dynamics) in-car VR content was reported to alleviate sickness symptoms for eight participants. However, two participants indicated that they experienced motion sickness. P15 noted, “Whenever a unique event happened, I’d focus on it and not feel as sick.” P17 added, “At first, I didn’t get what the cue meant, but after experiencing it a couple of times, I realized it’s a tool to inform about the driving situation, which helped with the motion sickness.” On the contrary, P8 stated, “Something dropped, and the sudden stop got me all anxious, making the motion sickness even worse.”. C1 (Baseline), C2 (Map), and C5 (Distraction) were predominantly negative in relation to sickness. These cues resulted in 5, 7, and 6 negative feedbacks, respectively. P15 commented on the C1 (Baseline), “Having nothing going on made my eyes tired, and I felt super nauseous.” P6 on the C2 (Map) said, “The more I got into it, the less sick I felt. However, the voice directions broke the immersion and made me feel nauseous.” Lastly, P27 remarked on the C5 (Distraction), “There were too many fish coming and going; it made my motion sickness way worse.”

Discussion of interview results

Based on the results, we can distinguish the distinct impacts of various cue types on the user experience within in-car VR content. The C3(Dynamic), characterized by high SA and presence, enhances users’ sense of immersion and simultaneously mitigates feelings of sickness. Additionally, cues with pronounced presence, namely the C4 (Game) and C5 (Distraction), amplify the immersive experience for users. Notably, the C4 (Game) also fosters a heightened sense of trust. However, the C2 (Map), despite bolstering user trust, was associated with diminished immersion and exacerbated sensations of sickness. While this cue directly conveys external information, it adversely affects the overall user experience.

6.3.2 Questionnaire results

Fig. 14

Total scores of NASA-TLX, SSQ, TiA, and FAM for each condition. Red stars indicate significant differences(*p < 0.05, **p < 0.01, ***p < 0.001; Error bars denote standard error (SE))

Table 4 Statistical analysis of Task Load scores across different conditions

NASA-TLX: The NASA-TLX questionnaire responses varied based on the cue type factor, as confirmed through RM ANOVA analysis (F(4, 116) = 4.466, p = 0.002, η² = 0.133). Post hoc tests showed that the C4 (Game) yielded the highest score compared to the other four conditions. Specifically, the score for C4 (Game) was statistically higher than for C1 (Baseline), C2 (Map), and C5 (Distraction). Although not significant, C3 (Dynamics) revealed a trend toward a higher score with a small effect size. This suggests that users exposed to the C4 (Game) in in-car VR content experience a higher workload compared to other conditions (Fig. 14 (top-left), Table 4).

SSQ: However, there was no statistically significant difference in SSQ questionnaire responses based on the cue type factor (F(2.709, 78.570) = 0.526, p = 0.648, η² = 0.018) as shown in Fig. 14 (top right).

TiA: Similarly, no statistical significance was found in the TiA responses (F(4, 108) = 0.892, p = 0.471, η² = 0.032), indicating that trust in automation was consistent across different cues in in-car VR content (Fig. 14 (bottom-left)).

FAM: Responses to the FAM questionnaire also showed no significant difference among conditions (F(4, 96) = 0.856, p = 0.493, η² = 0.034). This suggests that the anxiety experienced by participants was consistent irrespective of the cue type provided in the in-car VR content (Fig. 14 (bottom-right)).

Discussion on Questionnaire Results: Based on the questionnaires, C4 (Game) imposes a significant workload on in-car VR users, which is not evident with the other cues. While C4 (Game) may enhance immersion and trust, it also requires users to undertake an additional task, thereby increasing their perceived workload.

7 Discussion

7.1 Effect of cues with high SA

Participants who engaged with in-car VR content using SA cues from both C2 (Map) and C3 (Dynamics) demonstrated significantly enhanced SA. By contrast, other cues did not show a meaningful difference in SA compared to the baseline. This suggests that employing cues specifically designed to provide information about external situations in in-car VR content can offer participants a heightened SA experience. Our study expected that in-car VR content that facilitates a high SA experience can potentially address core issues inherent to in-car VR, such as poor sickness and immersion, as well as concerns over obscured peripheral vision, which increases reliance on vehicles and drivers, impacting trust. To evaluate the user experience of individuals playing in-car VR content by incorporating SA cues, specifically C2 (Map) and C3 (Dynamics), we used findings from the user study interviews.

Analysis of the trust ranking scores revealed that participants who experienced C2 (Map) exhibited a high level of trust. Qualitative insights gathered from interviews suggest that this trust is because of the transparent representation of the vehicle’s intended path, relaxed participants, and enhancement of their enjoyment. However, C2 (Map) exhibited poor immersion and presence. Distractions from navigation’s visual and auditory alerts inhibited participants from fully engaging with the content. Some participants even reported feelings of sickness attributed to this disturbance. Hence, while C2 (Map) directly delivers SA to the user and enhances trust in in-car VR content and the vehicle platform, it diminishes the user experience by prompting users to focus on the C2 (Map) rather than the content.

Conversely, C3 (Dynamics) had considerably different effects compared to C2 (Map), despite conveying similar external situation information to the user. An analysis of the IPQ results showed that users felt a significant improvement in their sense of presence in underwater content with C3 (Dynamics). Additionally, C3 (Dynamics) led to a significant elevation in immersion ranking scores. The distinction between the two cues arises from the way each operates—contrary to C2 (Map), C3 (Dynamics) incorporates elements consistent with underwater content to convey information about external situations. This approach affords users a heightened sense of presence and immersion. Feedback from interviews supported this finding, with participants feeling more immersed because of the integration of vehicle movements such as rotation and shaking into the content’s context. This design feature also mitigated participants’ sensations of sickness associated with the VR content. However, an analysis of the trust ranking scores revealed that participants had a diminished level of trust in the in-car VR contents. Some participants indicated that understanding the intent behind C3 (Dynamics) allowed them to anticipate vehicle movements, instilling a sense of safety. These findings imply that C3 (Dynamics) effectively conveys information regarding external situations without compromising user engagement with the content. Furthermore, when users fully comprehend the significance of the C3 (Dynamics) condition, issues related to sickness and trust can potentially be addressed. Thus, C3 (Dynamics) is perceived as a promising solution to address various challenges associated with in-car VR content and its platforms.

7.2 Effect of cues with high presence

Upon exposure to in-car VR content featuring C4 (Game), and C5 (Distraction), participants exhibited an increase in their perceived sense of presence. The cues from C4 (Game) and C5 (Distraction) shifted the focus from external situations to elements or stimuli within the content, providing deeper engagement. Specifically, in C4 (Game), based on a strategy by Wilson et al. (2023), a secondary task, such as collecting jewels, was used to divert users from the external environment. By contrast, Malloy and Milling (2010), and Bos (2015) designed C5 (Distraction) to immerse the user by providing rich visual and auditory stimuli at timings identified as demanding SA through a pilot study. It was hypothesized that these two types could potentially improve user experiences such as sickness, immersion, and trust.

C4 (Game) exhibited the highest immersion ranking score and achieved the second-highest IPQ score. Moreover, through interview feedback, a number of participants confirmed heightened engagement with the content during the execution of the game task. Users actively sought jewels and monitored their collection count, leading to the perception of time passing quickly. Such focused engagement with the content, coupled with a reduced awareness of the external environment, was found to be associated with an increased trust ranking scores. However, the prevalent feedback underscored sickness experiences during tasks. Participants experienced feelings of sickness attributed to oscillations from both their bodies and the HMD resulting from vehicular bumps, particularly while concentrating on content to locate jewels dispersed throughout the content map. Additionally, findings from the NASA-TLX assessment indicated that participants encountered a notably elevated workload, which can be attributed to the secondary task inherent to the C4 (Game). Although C4 (Game) enhanced presence, immersion, and trust, the secondary task introduced complexities, particularly in workload and sickness.

Concurrently, C5 (Distraction), C4 (Game), and C3 (Dynamics) were identified as significant contributors to enhancing presence and immersion among participants. An analysis of the immersion ranking scores verified elevated immersion levels among participants. Moreover, several interview responses indicated that C5 (Distraction) augmented their immersion within the in-car VR content. Rich visual and auditory stimuli provide users with immersion within the virtual environment. However, while the C5 (Distraction) aimed to improve the user experience by minimizing external environmental demands through stimuli during SA-demanding instances, there were no significant findings or feedback on improvements in trust or sickness. Thus, although C5 (Distraction) augmented the immersiveness of in-car VR content within the virtual environment, challenges surrounding trust and sickness remain. Future studies should prioritize refining cues to effectively address these challenges.

7.3 C3 (Dynamics) for in-car VR contents

The C3 (Dynamics) condition substantially elevated the SA and presence of in-car VR content. Based on findings from both a literature review and a pilot study, the design of C3 (Dynamics) was conceived to relay information about external situations to users at moments identified as desirable, with the intention of enhancing the user experience. However, unlike C2 (Map), C3 (Dynamics) offered information about external situations using elements used in underwater contexts, presenting them as contextual visual and auditory stimuli. It is believed that this design strategy made in-car VR content with C3 (Dynamics) capable of delivering high levels of SA and presence to the user. As observed with the C4 (Game), content with a high presence allows users to immerse themselves more deeply. This was also confirmed by the C3 (Dynamics), which displayed an immersion ranking score comparable to that of the C4 (Game). Feedback from participants also validated that the multimodal stimuli enabled deeper engagement with the content. Consequently, our methodology, which entailed designing and implementing C3 (Dynamics) to convey external situational information without compromising the content’s context, stands as a pivotal strategy in developing immersive in-car VR content with which users can fully engage.

However, any quantitative statistical analysis showed a significant improvement in the sickness and trust experience of in-car VR content. Yet, from interview feedback, it was evident that the sickness and trust experiences of some participants improved because of C3 (Dynamics). These individuals fully comprehended the intention of C3 (Dynamics)’s visual and auditory stimuli. They were able to perceive external situations sufficiently through C3 (Dynamics), immersing themselves in the various effects without disrupting the progression context of the content. Furthermore, they felt that the characteristics of C3 (Dynamics) alleviated their feelings of sickness and distrust. However, some participants initially misunderstood the intent of C3 (Dynamics) but reported an enhancement in the user experience as they came to understand the cue’s intended purpose. For instance, P3 mentioned, “At first, I didn’t realize the content was syncing with external situations. But as I interacted more, I learned how each device worked. This reduced motion sickness and made me feel more immersed.” Similarly, P17 noted, “Initially, I was confused, but after a couple of tries, I figured out that the device was tracking driving conditions, which helped minimize motion sickness. Additional effects, such as falling rocks, further enhanced the immersive experience.”

From the findings, it becomes evident that there is a pressing need to consider an accessible design for the C3 (Dynamics) condition, ensuring that users can effortlessly and comprehensively grasp its implications. One proposed method to enhance the transparency in C3 (Dynamics) is the incorporation of a preliminary step before the content initiates, elucidating the intent of each stimulus. Alternatively, an in-content manual detailing upcoming events can also be provided. It is anticipated that by addressing these improvements and refining C3 (Dynamics), the content’s SA can be augmented, resolving issues related to sickness and trust. Consequently, this would facilitate a more profound and immersive user experience, enabling users to engage more actively in the virtual environment.

7.4 Limitation and future works

In this study, we investigated the driving scenarios that induce discomfort in passengers using in-car VR content. We assessed the effect of SA cues or presence-enhancing cues on improving the user experience with in-car VR. We provided directions for future studies for overcoming current limitations.

First, concerning the driving scenarios that induce discomfort, we addressed physical situations that might arise on roads but did not examine events triggered by traffic conditions (e.g., lane changes, sudden braking, and traffic signals). Understanding passengers’ needs in various driving scenarios, including those caused by other vehicles or pedestrians, and conducting investigations using the suggested cues or exploring new cues, are necessary.

Second, in the scenario in which SA cues were provided, the C3 (Dynamics) strategy was not always immediately intuitive and did not accomplish the purpose of the provided visual and auditory stimuli. This phenomenon was in contrast to that in C2 (Map). Therefore, design improvements that enhance user comprehension from the outset are critical for revealing areas for enhancement. Future studies can address these challenges by exploring user-friendly interface designs for C3 (Dynamics) through the integration of preliminary content tutorials or in-content guides for elucidating the expected outcomes of the C3 (Dynamics) cue.

In conditions, such as C4 (Game) and C5 (Distraction), for augmenting presence and immersion, some participants reported motion sickness because of a discrepancy between the visual experience and physical sensation of vehicle movement when engaging in tasks such as searching for scattered gems or identifying numerous fish. This phenomenon indicates the necessity of concurrently providing SA cues with presence-enhancing cues. Structuring content elements to incorporate interaction could enhance engagement with the content scenario. Incorporating interaction with the intended SA cue could elevate focus on the SA cue, which could effectively convey SA information.

Finally, by expanding beyond immersive content to various VR applications (e.g., gaming, conducting remote meetings, editing documents, and viewing media), novel opportunities to enhance SA and immersion can be identified, and a comprehension of how different tasks influence the user experience of in-car VR can be achieved.

8 Conclusion

In this study, we compared the user experience of in-car VR content with five different cues. Thirty participants wore an HMD and sat in the passenger seat while the vehicle drove on real roads. They responded to questionnaires and interviews regarding SA, presence, immersion, motion sickness, trust, workload, and anxiety. C2 (Map), which provides an external SA cue, showed a high level of trust but lower immersion and less pronounced motion sickness. By contrast, C3 (Dynamics), by offering SA cues through diegetic cues, enhanced immersion in the content and indicated the potential for reducing motion sickness. This suggests that SA cues that enhance immersion in the in-car VR content are generated even without directly providing SA information. In the study, we demonstrated how SA cues can improve the usability of in-car VR content, subsequently enhancing the user experience. We also discussed several issues that developers should consider in the gradually evolving field of in-car VR.