Challenges in passenger use of mixed reality headsets in cars and other transportation

Abstract

This paper examines key challenges in supporting passenger use of augmented and virtual reality headsets in transit. These headsets will allow passengers to break free from the restraints of physical displays placed in constrained environments such as cars, trains and planes. Moreover, they have the potential to allow passengers to make better use of their time by making travel more productive and enjoyable, supporting both privacy and immersion. However, there are significant barriers to headset usage by passengers in transit contexts. These barriers range from impediments that would entirely prevent safe usage and function (e.g. motion sickness) to those that might impair their adoption (e.g. social acceptability). We identify the key challenges that need to be overcome and discuss the necessary resolutions and research required to facilitate adoption and realize the potential advantages of using mixed reality headsets in transit.

Introduction

Advances in transportation are such that there will be a significant conversion from drivers to passengers over the coming years, i.e. in transit but not necessarily responsible for the continuous navigation or control of a vehicle. This is due both to the increasing availability/usage of public transport and the impending adoption of autonomous vehicles. As a consequence, the not inconsiderable time spent across the population as passengers is expected to continue to rise. A recent report showed that long commutes in the UK have risen (Trades Union Congress 2015) to the extent that UK car journeys typically last $\sim$ 22 min (Department for Transport 2016) and commutes last $\sim$ 55 min (Press Association 2015), whilst in the USA drivers spend $\sim$ 56 min per day in transit (U.S. Department of Transportation 2009). For flights, Heathrow alone reported 78 million passengers in 2017 (Heathrow 2017), whilst train journeys in the UK have doubled since the 1990s, with 145 million long-distance journeys in 2018 (Office of Rail 2018). And, more broadly, 3.7 million workers in the UK travel more than 2 h every weekday (TUC 2016). In effect, a considerable portion of our lives is spent as passengers, undergoing journeys that can be perceived as repetitive and whose duration is often considered wasted time (Gardner and Abraham 2007; Watts and Urry 2008).

Given this, an increasingly important question is how best to support passengers to make the most of this travel time, regardless of the mode of transportation they undertake. Observations of passenger behaviours have tended to reveal that modern passengers now alternate between the expected, productive activities—reading, watching videos—and looking ahead, or out of a window (Russell et al. 2011), with such behaviours also observed in autonomous driving scenarios (Hecht et al. 2020). The conversion of drivers to passengers, fuelled in part by autonomous car adoption, has the potential to better enable passengers to perform these productive activities, with considerable benefits for society economically.

However, there are open questions regarding how technology can best assist passengers in making the most of their time in transit. For example, how should displays be integrated or embedded into the given mode of transit: via dashboard surfaces (Ng and Brewster 2016; Ng et al. 2017) or seatback displays (Prince 2014), head-up displays (HUDs) (Pauzie 2015), on (Häuslschmid et al. 2015) or over (Matt Kamen 2016) windows, etc.? And what are the consequences of this ubiquity of displays? It has been noted in air travel for example that “already there are too many screens in the plane with monitors on the seats and passengers bringing on their range of personal devices. When people are trying to rest it is already difficult with the glare of all these devices. It would be good to focus some work on individual private spaces” (Frangakis et al. 2014). There are also issues regarding how interaction should be supported in transit (Ng and Brewster 2016; Shakeri et al. 2016), whilst, most fundamentally of all, there is the “elephant in the room” (Diels et al. 2016) of motion sickness which, particularly with road-based transit, has the potential to significantly impede passenger adoption and usage of technology (McGill et al. 2017). Looking out of windows is a common, necessary and often enjoyable way to minimize motion sickness—albeit at the expense of disrupting any non-travel-related activities. There is a pressing need to examine such issues, given that the act of driving helps to prevent motion sickness, and viewing stable content (e.g. reading a book, watching a movie on a tablet) in a motion environment can trigger motion sickness symptoms (Diels and Bos 2016, 2015; Sivak and Schoettle 2015). And, despite the fact that autonomous cars will allow for a radical redesign of the interior (e.g. rear-facing front seats, windows becoming displays (Diels and Bos 2016)), as with other modes of transport such as planes, trains or boats, passengers will still fundamentally perceive themselves as being in a constrained, and often overly familiar, space for prolonged periods of time during their journey.

Against the context of passenger adoption of technology, we suggest that there is sufficient justification to explore the use of mixed reality (MR) headsets, encompassing both virtual reality (VR) and augmented reality (AR), for use by passengers across a variety of transportation, with a particular focus on in-car and in-flight usage (Fig. 1). Assuming that autonomous vehicles reach the level of full autonomy (SAE Levels 4/5 (International 2016)), trust in said vehicles is sufficient (Häuslschmid et al. 2017), and comfort/security concerns can be assuaged, usage of MR headsets by all passengers in transit becomes a feasible possibility. MR headsets offer a number of benefits over existing in-transit displays. They have the potential to both improve the usability and ergonomics of existing passenger activities by comfortably rendering and placing 2D and 3D content with depth anywhere around the passenger. The nature of the medium is such that content is innately personal and private, unless shared through software. And the headsets themselves are low power, with consumption similar to that of a smartphone, growing based on the fidelity of rendering and consequent graphics processing unit required. VR headsets in particular also allow users to appear to entirely escape the confines of the vehicle and become present and immersed in a different, entirely virtual, environment (often referred to as “place illusion”, a component of presence (Slater 2009)). Notably, because MR headsets have the capability to track head orientation and render content on that basis, they also potentially have complete control over how motion is visually perceived. This has significant implications for combating motion sickness (McGill et al. 2017), commonly resulting from a mismatch between how motion is physically (e.g. via the vestibular system) and visually perceived (Reason and Brand 1975; Zhang et al. 2016). Consequently, we might envisage the passenger experience improving in terms of:

• Productivity The user sits down in their autonomous vehicle for the daily commute. Their AR headset renders a wide virtual workspace around them, allowing them to begin work immediately, and look forward to leaving work earlier as a result.

• Entertainment Gazing at the real landscape in front of them, the train passenger’s AR headset highlights landmarks of note and generates new games and experiences out of the available landscape and location, e.g. rendering characters running alongside the vehicle in a high-speed platformer game.

• Isolation As the seatbelt sign turns off after take-off, the passenger puts on their VR headset, entirely occluding the sights and sounds of reality, finding themselves in a relaxing virtual home cinema, modelled after their own home for comfort.

Fig. 1

Two use cases of MR displays in transit of particular note are (left) in-car (McGill et al. 2017) and (right) in-flight (Williamson et al. 2019), as both use cases have the potential to feature private/secure surroundings with journeys of long durations

However, there are a number of challenges that could impede the adoption and usage of MR headsets in transit, and consequently the realization of such visions. This paper outlines what we consider to be the most significant of these, based on our previous research regarding VR headset usage in-car and a review of the emerging literature intersecting the domains of VR/AR headsets, in-car interactions and human factors challenges in autonomous cars. Firstly, we briefly discuss the problems with existing in-vehicle display technology that justify the deployment of MR headsets in transportation. Secondly, we consider the most pressing problems regarding supporting MR headsets in transit, which are split into two categories:

• Functional impediments to MR usage in-motion These are fundamental problems that effectively rule out safe and proper passenger usage of headsets currently, such as headsets being unable to retain a forward bearing in-motion, being untested with respect to crashworthiness and being likely to induce motion sickness.

• Impediments to acceptance/adoption Assuming MR headsets can function correctly in-motion, and are safe to use, barriers to their acceptance and adoption by passengers should be considered. For example, the social acceptability of use across different forms of transport would seem likely to vary significantly, dictated by issues such as managing awareness (of fellow passengers and the travel environment), ensuring passengers are still approachable and accessible, etc. Conversely, providing compelling reasons to use headsets in transit might positively influence adoption, e.g. gamified experiences that rely on motion could provide more exciting and enjoyable journeys. And achieving/maintaining parity with headset features that users may come to know as standard, such as telepresence, will be important to meet existing user expectations of the technology.

In this paper, we focus predominantly on passenger usage of MR headsets for tasks other than driving. We explicitly exclude driver-assistive AR (Gabbard et al. 2014; Kun et al. 2016)—whilst some of the problems discussed are applicable to both passenger and driver MR headset use, such use cases bring with them a host of new considerations regarding distraction, trust, communication and safety and should be considered as separate from the passenger MR headset use case. We also concentrate on headsets (both auditory and visual) rather than integrated or mobile displays, for the aforementioned potential benefits (privacy, ergonomics, etc.) and because they offer the most transportation-agnostic way of enabling MR passenger experiences. If, as anticipated, such headsets see further adoption by consumers for everyday usage (e.g. immersive VR for entertainment, or wearable AR for mobile spatial computing), it is reasonable to consider how we can support and take advantage of these headsets when they are inevitably transposed to transit contexts by users. Finally, we also exclude discussion of the network infrastructure required to deliver live/streamed mobile MR experiences, with a number of challenges identified in the mobile delivery of low-latency, high-fidelity MR content (Mangiante et al. 2017; Elbamby et al. 2018). This problem will likely be resolved by the advent of 5G networks (Orlosky et al. 2017), and the usage of offline content means we do not consider this to be a significant impediment to initial adoption.

Broadly, the intention of this paper is to outline both why support for MR headsets in transit should be considered, and in what areas concerted research effort across academia (with an emphasis on usability and HCI) and industry is required to bring about safe and compelling MR headset experiences for passengers, regardless of the mode of transport.

The case for passenger MR headsets

The problems with in-vehicle displays

Phones, tablets and laptops are a common sight inside planes, trains, boats, buses and cars, offering passengers a distraction from the sights and sounds of their journey. This has long been a necessary feature of travel, with entertainment providing a “diversion from speed and the risk of the catastrophic accident via screens”, first noted during “nineteenth-century railroad journey(s), in which the new practice of ‘panoramic vision’ and the reading of newspapers by passengers onboard railcars served as a ‘stimulus shield”’ (Groening 2013). As a more contemporary example, in-flight entertainment has been suggested to act as “an intermediary, screening out the fact of flight and the events of travel...crucial to keeping passengers calm, occupied and content” (Groening 2013) and has been noted to significantly improve the perceived comfort of passengers (Ahmadpour et al. 2014a; Patel and D’Cruz 2017).

For the modern passenger, in addition to personal devices, any number of in-built displays are often available, e.g. dashboard displays in-car and seatback displays for passengers seated in the rear (Wilfinger et al. 2011). Recent advancements include projection-based AR HUDs (Pauzie 2015) and windows (Rao et al. 2014a; Häkkilä et al. 2014; Haeuslschmid et al. 2016; Rao et al. 2014b), whilst technologies such as flexible OLED displays mean that every surface could effectively become a display, as demonstrated in concept cars (Mercedes-Benz 2016). The necessity of windows may even be questioned, and they may just become displays, occluding outside visibility for presentation (Matt Kamen 2016). (See Fig. 2 for examples.) This technological integration and adoption has led to the modern passenger experience of distraction being very different to that of railroad passenger journeys a century ago. Now, passengers can find themselves ensconced in a “techno-cocoon” with technology acting as a “sensory filter...crucial for sensory privacy and exertion of control” (Groening 2016).

Fig. 2

Examples of different in-vehicle display concepts. Top left: Seatback displays (Thales 2018); top middle: Ford windshield movie screen; top right: Toyoto AR windows (Toyota 2011); bottom: Mercedes-Benz concept car with every surface as a display (Mercedes-Benz 2016)

Indeed, as Groening (2016) notes, “at least since the 1970s, passengers have sought more forms of separation between themselves and those other passengers perceived as ‘undesirable’ ”. However, all of the aforementioned displays fall short of this underlying aim of separation. This is because they are effectively integrated into the shared physical environment, and consequently they share common problems and drawbacks. They are often limited in terms of size, and thus immersion (Cummings et al. 2015). Privacy is rarely assured, with displays frequently gaze-accessible to other passengers. They are subject to glare and reflections from the ever-changing lighting of the outside environment and can require a gaze angle that, at the very least, can be sub-optimal in terms of comfort (e.g. staring downwards, or staring at one fixed place continually for long periods, noted to cause neck problems due to a lack of variety of head movement (Farias Zuniga and Côté 2017)). At worst, they are also more likely to induce motion sickness, termed nauseogenic visual displays (Golding and Gresty 2015). Given passengers often require visual awareness of the motion of the car to avoid motion sickness (Elbanhawi et al. 2015; Diels and Bos 2015), some view of the outside world or the motion of the vehicle may be necessary, with any restrictions of the view having further consequences for motion sickness. Fundamentally, these displays have to work around, and within, their physical context, whilst passengers still perceive themselves as being in the constrained and repetitive space of a car, plane, train or bus.

The benefits of MR headsets: immersion, limitless display spaces, and visual perception of motion

Conversely, MR headsets can potentially overcome many of these problems. They are unrivalled in terms of immersion/presence (Cummings et al. 2015), with privacy dictated by software constraints rather than physical visibility. Occlusion issues are no longer relevant as any view can be presented in the headset, whilst interactions can move with the user (given the headset is subject to the same oscillations as the body). Healthy, comfortable gaze angles can be enforced as content can be displayed anywhere around the user, and moved at any time. New possibilities for interaction (e.g. via gaze (Lucero and Vetek 2014) or direct touch (Chan et al. 2010)) and communication (e.g. telepresence where those you are addressing appear in your local environment (McGill et al. 2016; Orts-Escolano et al. 2016)) can be supported. And stereoscopic MR headsets can also render content with depth.

The VR Hyperspace EU project, which finished prior to the advent of new, cheap, consumer VR headsets such as the Oculus Rift, noted the benefits of VR usage in transit, with their stated aim being to “enhance the passenger comfort through...(the) adoption of virtual and mixed reality technologies in the future air cabin” (Cappitelli et al. 2014). More recently, airlines have also tested VR headsets for use in-flight (Air France 2017; Qantas 2015), with benefits even being shown regarding safety knowledge transfer regarding how to wear an aircraft life preserver (Chittaro et al. 2018). There have also recently been a number of instances of industry explorations of VR headset usage in both planes (Kuchera 2015; Gulliver 2017; Holly 2017) and autonomous cars, such as by Renault (Dent 2017) and Apple (Rober et al. 2018). As a blogger for The Economist noted: “virtual reality headsets on planes mean we can isolate ourselves from irritating cabin-mates” (Gulliver 2017).

MR headsets allow for immersive gaming and entertainment experiences (Cummings et al. 2015) and will inevitably also support general productivity applications (McGill et al. 2015). But they can also alter the passenger experience in more subtle ways. The VR Hyperspace project previously noted that such technology could change the perception of self and space, improving passenger experience and comfort through new virtual “surroundings, real imagery and live flight data in addition to fantasy environments” (Frangakis et al. 2014). Indeed, it has been noted that:

Virtual environments can fully or partially distract people from sources of discomfort, becoming more effective when they are interesting. They are also more effective at distracting people from discomfort caused by restricted space than noise disturbances (Lewis et al. 2016)

MR headsets can also better take into account passenger attitudes towards their environment. For example, Ahmadpour et al. (2016) classified these attitudes in terms of: adjusting, avoiding, approaching and shielding, linked to “passengers’ concerns for control, privacy, social connectedness and/or social tolerance”. MR experiences can facilitate many of these behaviours. For example, they allow content to be positioned optimally based on user comfort and ergonomics. They can avoid violations of personal space caused by other’s digital activity by having a private virtual personal space. And they can shield passengers from undesirable behaviours such as auditory or visual noise. Creating the perception of a more personal and private space, immune to invasion by others, is a significant benefit of MR headsets (Lewis et al. 2017). On this basis, we could envisage that an MR headset user/passenger might, for example, retreat to their private, virtual workspace, with a similar layout and interactions in-car as in their workplace.

The case against passenger MR headsets

However, for such a scenario to become a reality, a number of problems must first be overcome. If we consider existing consumer MR headsets outside of their use in transit, they still have some notable limitations. In the case of VR headsets, use is limited with respect to field of view (typically ranging from $\sim$ 90$^{\circ }$ to $\sim$ 130$^{\circ }$ in current consumer models) and fidelity, comfort and weight, especially on-the-move where mobile headsets are often powered by smartphones. For AR headsets, some of these issues are more pressing, for example the Microsoft Hololens 2¹ currently supports only a $\sim$ 70$^{\circ }$ horizontal field of view. However, we can reasonably make the assumption that lightweight and hi-fidelity MR headsets will inevitably become a consumer reality over the coming years, given the recent leaps made in terms of hardware, software and processing power. For example, consider the advances made between the release of the Oculus Rift DK1 in 2013 and the Oculus Quest² in 2019. Over the space of 6 years, there were significant leaps in resolution, refresh rate, positional tracking, mobile use, fidelity, tracked controller interaction, hand tracking, etc.

Assuming that mobile consumer headsets reach the anticipated level of maturity and utility, consideration should be given towards anticipating the future impediments to the use of MR headsets in transit, so they can be dealt with in parallel with the development and mass adoption of these headsets. The VR Hyperspace project offered the most comprehensive discussion of use in transit thus far (Frangakis et al. 2014). They suggested that the most problematic barriers to the use of VR in-flight were those of:

• Cost To passengers, transportation companies and manufacturers

• Reluctance to be immersed in VR For example, due to mistrust in technology, being unaware of safety-related conditions, etc.

• Standardization In terms of headset platforms and interoperability with vehicle environments

• Security and privacy For example, supporting the sharing of personal data to drive VR experiences

Our paper builds upon this work. Based on our review of the nascent literature on passenger MR experiences, and our prior research into VR headset usage by passengers both in-car (McGill et al. 2017) and in-flight (Williamson et al. 2019), we identify and outline the most significant research challenges, classified in terms of being either functional impediments to use, or design challenges that, when overcome, might increase acceptance and adoption of these new technologies.

Functional impediments to MR usage in-motion

By functional impediments, we refer to challenges and problems that would prevent or limit normal usage of MR headsets in transit. We discuss three functional impediments that we suggest require concerted effort and consideration to overcome: how existing IMU-based head orientation tracking is confounded by the presence of external motion; the unknown safety of MR headsets in the event of a crash/airbag impact; and how discrepancies between what motion is physically perceived (e.g. via the vestibular system) and visually perceived (e.g. being present in a stationary virtual room) will lead to the onset of motion sickness. For each of these challenges, we discuss the state of the art in terms of solutions, and what further research is necessary to overcome this problem. We also refer transportation in terms of the motion experienced by the passenger, grouped into three common types:

• Motion type 1 Stable/constant velocity, infrequent vehicle orientation changes, little-to-no oscillations, e.g. large commercial planes or cruise ships.

• Motion type 2 Frequent orientation changes, e.g. autonomous car journeys on motorways, intercity trains.

• Motion type 3 Frequent vehicle velocity and orientation changes with oscillations, e.g. autonomous car journeys in cities, inner city trains/buses.

Challenge 1: maintaining a forward bearing in a moving world

Most affected: motion type 2/3 Transportation that involves frequent and/or significant changes in orientation and acceleration (e.g. cars, buses, boats).

To understand the difficulties in performing head tracking in transit, first we must discuss how MR headsets typically support rotational and positional tracking. This is achieved through IMU-based sensor fusion of a gyroscope and accelerometer, captured at a high sampling rate ($\sim$ 1000 Hz in the latest headsets) and low latency for dead reckoning (LaValle et al. 2014). To compensate for sensor bias/drift over time, additional sensing is used for frequent corrections, so that what the user perceived as forward when they started using the headset is still in the same physical direction after prolonged use. For simple rotationally tracked headsets (e.g. the Gear VR), the magnetometer is responsible for providing this correction factor, offering a constant bearing for magnetic north. For positionally tracked headsets such as the Oculus Quest or HTC Vive³ and their successors, optical tracking is used to provide this correction. There are two common approaches to this end, referred to as “inside-out” and “outside-in” tracking. Inside-out tracking refers to headset-based sensing for determining the position in the world. In the case of the HTC Vive and successors such as the Vive Focus⁴, “Lighthouse” beacons in the physical environment broadcast pulses of IR light which the headset detects (Buckley 2015). However, recent consumer headsets such as the Oculus Quest now typically use SLAM-type (Durrant-Whyte and Bailey 2006) inside-out tracking enabled by depth cameras integrated into the headsets, using computer vision to track the surrounding environment, and consequently the position of the headset within said environment. Outside-in tracking typically refers to tracking systems where the corrective sensing is embedded in the environment. The Oculus Constellation system for example uses multiple external IR cameras with IR emitters embedded in the tracked objects (Feltham 2015). Regardless of the tracking technology used, positionally tracked headsets have the benefit of knowing their absolute position in 3D space and adapting their presentation accordingly, minimizing the discrepancy between what is visually and physically perceived, and consequently minimizing simulator sickness (Davis et al. 2014) whilst better facilitating presence (Cummings et al. 2015).

Problematically, due to the current reliance on IMU-based sensor fusion, both rotationally and positionally tracked headsets share a common problem when in-motion: any in-built IMU sensing is no longer detecting head movement alone⁵. Instead, it is now detecting a combination of user head movement and vehicle accelerations/rotations (Fig. 3), and then applying periodic corrections based on additional, usually optical, sensing. In the case of rotationally tracked headsets (e.g. Samsung GearVR), they will lose track of the forward bearing of the user, meaning that the MR headset will not be able to maintain a stable focus on 360$^{\circ }$ content under motion, with the user’s view turning with the vehicle. In the case of positionally tracked headsets that rely on additional optical tracking, external motion will lead to the user experiencing judder as the corrective optical sensing contradicts the IMU sensing. And future MR headsets are likely to exhibit the same problems so long as they rely on IMU-based sensor fusion and/or use corrective tracking technology that has not been adapted for use in the uniquely problematic environment of a moving vehicle.

Fig. 3

The MR headset undergoes translations (indicated by the blue/green/red arrows indicating translations in x/y/z) and rotations (indicated by the blue/green/red circles) relative to the interior of the vehicle. However, the vehicle also undergoes translations and rotations relative to the world, which may also be detected/conflated by the headset sensing (colour figure online)

Resolving this challenge

Consequently, providing a general-purpose solution for enabling rotational and/or positional tracking for consumers across a variety of motion environments, from planes and trains to autonomous vehicles, represents a significant research problem, requiring additional sensing to either correct the headset inertial sensors and provide a stable forward bearing, or to replace the reliance on IMU-based inertial sensing for tracking headset orientation. Figure 4 details the potential sensing solutions we envisage.

Fig. 4

Existing (delineated in red) and anticipated permutations of head-mounted displays (HMDs) and vehicle sensing for providing MR passenger experiences across different types of transportation

Examples of correcting the inertial sensors can be seen in both McGill et al. (2017) and Hock et al. (2017), where a car-mounted gyroscope was used to allow car orientation changes to be subtracted from a rotationally tracked headset’s orientation. However, in this case, gyroscopic drift still occurs, so any such vehicle-based corrective IMU sensing would still require periodic corrections itself, e.g. the user explicitly instructing the headset to reorient when facing forward, or using additional sensing such as GPS or optical tracking as a means to measure the exhibited drift.

Ideally, we might perform such corrections on positionally tracked headsets, given the necessity of positional tracking for avoiding simulator sickness. As Fig. 4 details, there are a number of possible permutations of sensing that can provide support for all three of the motion types discussed, and it is by no means clear what the optimal solution might be here for any given transportation. This is because positionally tracked headsets face additional challenges in-motion. For outside-in headsets, cameras or sensors need to be mounted in the vehicle environment and need to be compatible for use in-motion. For example, HTC Vive Lighthouse beacons (both V1 and V2) cannot yet be used in-motion, relying on a hard drive motor (Skarredghost 2017) that shuts down to prevent damage when external motion is detected—a solid state equivalent would be required. Passenger seating and movement may pose challenges in terms of occlusion and lighting, whilst such solutions would only be viable for a subset of transportation where such technology could be pre-installed or quickly deployed.

Inside-out headsets might appear preferable then, requiring no hardware additions to the external environment. However, depth camera-based optical tracking solutions come with their own technical challenges, being particularly suited to enclosed and stable environments with little-to-no outside visibility such as in planes. This is in contrast to environments such as car interiors that feature a large proportion of refractive and translucent surfaces such as windows (problematic for any IR-based depth camera), and a constantly shifting visual environment (problematic for binocular depth imaging and tracking features in the world). In effect, any visible external motion (e.g. moving landscape through windows) would be likely to impair tracking, requiring such headsets to be able to fixate on known, stable interior features for tracking. Nonetheless, we could perhaps envisage a peripheral which combined a single optical tracking camera or optically trackable marker and an IMU, which could be temporarily mounted on the vehicle within visibility of the MR-headset-wearing passenger. Such a device could facilitate positional tracking in a range of environments and provide corrective data regarding vehicle orientation.

At the extreme, we could envisage future tracking solutions that no longer rely on IMUs. For example, instead of headset optical tracking solutions providing corrective pose updates and positional tracking, they could be used to exclusively provide headset orientation data. However, on top of the previously discussed problems with using optical tracking in-motion, removing the reliance on IMUs poses significant technical challenges regarding latency, accuracy and processing requirements for generating the headset pose, particularly when considering Timewarp-type solutions that require high-frequency/low-latency sensor data to function (Antonov 2015).

Much as with the variety of tracking technology in current consumer MR headsets, this is a research problem with a number of viable potential solutions, each potentially better suited to different forms of transport and different use cases. For example, if we consider the case of autonomous cars, research collaborations will likely be required between headset manufacturers and the automotive industry to arrive at car designs which provide adequate support for this use case. This may be in the form of designing standards for sharing car-bearing telemetry at low latency, integrating IMU or positional tracking sensors or creating car interiors that are conducive to being optically tracked by passenger headsets (e.g. integrated IR LEDs, markers or other discrete optically trackable features). Conversely, for usage in planes, given plane bearing changes are relatively infrequent and head movement is largely restricted to rotations only, it may be that software-only solutions (such as periodic user-driven re-calibration/zeroing) or initial versions of consumer inside-out tracking (capable of tracking the plane interior given the lack of windows/consistent lighting conditions) may be sufficient to enable widespread usage with little-to-no disruption.

Challenge 2: Physical crash safety

Most affected Transportation where recoverable crashes can occur without warning (e.g. cars), as opposed to where incidents may be forewarned and the headset removed (e.g. planes).

In the event of a crash, the seat belt restrains the passenger’s body whilst, in cars at least, the airbag is deployed to slow down the change in momentum of the head. However, use of an MR headset during a crash may impact the effectiveness of these safety measures. Firstly, MR headsets increase the mass of the users head (e.g. VR headsets typically weight around 0.5kg (The 360 Guy 2019)), with the average human head weighing approximately 5kg (Gekhman 2006), meaning a $\sim$ 10% increase), and consequently will increase the force exerted on the neck during a crash. The consequences of this (e.g. regarding whiplash incidence) are yet to be established. Secondly, the user may be denied any warning cues or anticipatory information regarding there being a crash imminent, and thus will be unlikely to brace safely. Thirdly, in cases where an airbag is present, the airbag will inevitably make contact with their protruding headset. At this point, the headset could break, resulting in broken glass from the lenses and display, plastic debris, and ruptured batteries in the case of mobile headsets. Or it could remain intact, applying the additional impact force resultant from the crash onto a smaller surface area where the headset makes contact with the wearer’s face.

Resolving this challenge

The deformability characteristics of current consumer headsets are untested and do not appear to be considered by manufacturers currently. Moreover, the consequences of such an impact are unknown. Whilst this situation is understandable, given the passenger use case has yet to see significant adoption or commercial endorsement, this point will need to be addressed to allow for safe, legal usage of MR headsets by passengers. There is a body of research examining the risks of eyewear such as glasses being worn during such events. Notably, whilst the additional risk encountered was small (Koisaari et al. 2017; Tervon and Sulander 2014), the severity of individual cases (Tsuda et al. 1999) combined with the dimensions, materials and form factors of current MR headsets suggests that crash safety should be evaluated for any headset where usage in transit is likely, such that users are at least made aware of any discovered risks. And, in time, it could be expected that consideration be given to crashworthiness at the design stage of headsets, acknowledging their potential usage in transit and safeguarding against being worn during a crash.

Challenge 3: motion sickness

Most affected: motion type 2/3 in VR Transportation that involves frequent and/or significant changes in orientation, acceleration or oscillation (e.g. cars, buses, boats).

The problem of motion sickness has remained ever-present in transportation. Whilst the fundamental cause is unknown, there are a number of theories that have helped to expand our understanding of the varied pathways causing motion sickness. For example, it is commonly theorized that a sensory mismatch between what the vestibular and the visual system perceive results in motion sickness (Reason and Brand 1975; Zhang et al. 2016). Others suggest that discrepancies in determining the subjective vertical (i.e. gravity) (Bles et al. 1998), changes in acceleration (Sawabe et al. 2017), conflict between the Canal-Otolith systems (Guedry and Benson 1978), lateral oscillations (Hosseini and Farahani 2015), postural instability (Riccio and Stoffregen 1991) and sway (Owen et al. 1998) all contribute to motion sickness.

Regardless of the underlying causes, if we examine road transportation in particular, autonomous cars will likely increase motion sickness incidence (Diels and Bos 2016, 2015; Sivak and Schoettle 2015). This is due to increased behaviours that induce motion sickness, for example increased use of displays and devices; changes to seating, with concept cars often proposing front seats that rotate to face backwards to form a more social interior; and the visibility of the external environment (Elbanhawi et al. 2015; Diels and Bos 2015), leading to increased sensory mismatch. In addition, the act of driving itself provides anticipatory cues which prevent motion sickness from arising in drivers (Bertolini and Straumann 2016; Golding and Gresty 2015; Rolnick and Lubow 1991), so when drivers become passengers more people will become sick. As Diels says:

...the use cases that are being envisaged for automated driving are also those we know to lead to increased levels of car sickness (Diels 2014)

MR headsets also have the potential to contribute to new forms of sensory mismatch, particularly in the case of VR headsets where reality is entirely occluded (McGill et al. 2017). Consider playing a VR game where the player controls their virtual movement independent of that of the vehicle. This could result in conveying no motion visually when the vehicle is moving, conveying motion visually that is entirely different to what is physically perceived (e.g. the car turns right but in the VR scene the view turns left) or even conveying motion which matches the physically perceived motion but at a different magnitude. The effects of such circumstances are largely unknown, but it would seem likely that, for some users, such usage will present a new test of their resilience to motion sickness, a consequence of experiencing stable/moving virtual content that may or may not be aligned with vehicle motion in reality.

Resolving this challenge

Notably, headsets also have the potential to contribute towards both understanding the causes of motion sickness and fixing the sensory mismatches that result in said sickness. As McGill et al. demonstrated (McGill et al. 2017), if the motion that is visually conveyed via the headset is congruent with what is physically perceived, motion sickness will be minimized, and it may be possible to blend visual motion cues with other stable content allowing for a general-purpose presentation of motion for any content type. Headsets can also visualize unseen motions, for example showing orientation changes when below decks on passenger ferries/cruise liners (Carter et al. 2018a; Stevens and Butkiewicz 2019)

In addition, if the vehicle is moving uniformly (e.g. no orientation changes, constant velocity), such as in an aircraft, motion sickness may not arise (Wienrich et al. 2017). Provoking anticipatory responses to motion through vection illusion has also shown some promise (Sawabe et al. 2017). And Soyka et al. incorporated motion into VR in the form of a magic carpet ride, finding that “brief exposure to turbulent motions [does] not get participants sick”, suggesting that VR use in-flight may not pose the same motion sickness risks as in other modes of transport (Soyka et al. 2015). VR has even been used to reduce seasickness (Carter et al. 2018b) by providing passengers below decks with a visual awareness of the orientation changes of the boat.

Head-worn equipment also suggests new possibilities for deploying vestibular counter-stimulation or desensitization, for example through Galvanic Vestibular Stimulation (Cevette et al. 2012), in conjunction with visual manipulations to reduce motion sickness. Moreover, entirely virtual displays can be presented anywhere around the user, not only potentially maximizing comfort but also minimizing car sickness, given that positioning in-car displays at eye height has been shown to have a significantly beneficial effect to motion sickness (Kuiper et al. 2018), in part due to the head being aligned with gravity.

We suggest there are significant open research questions regarding:

• Visual motion cues What motions need to be conveyed (orientation, acceleration, velocity, etc.)? What are the design parameters (e.g. perceived magnitude of motion (Wilson et al. 2018b), portraying accelerations versus velocity, the necessary perception of optic flow (Diels 2008; Redlick et al. 2001))? How can visual motion cues be presented in virtual experiences, either interleaved with content or shown in a content-agnostic manner (e.g. as with the peripheral motion cue examined in (McGill et al. 2017))?

• Anticipatory motion cues For example displaying impending motion (Karjanto et al. 2018) or tricking the MR passenger into performing anticipatory motion actions.

• When motion cues should be presented For example based on user preferences or motion thresholds in an attempt to minimize the visual disruption or distraction the cues might cause.

• The importance of rest frames Having previously been shown to be important for simulator sickness as a result of vection (Hock et al. 2017; LaViola 2000) (see Sect. 5.4 for examples of usage for immersion), what role should rest frames play in perceiving real-world motion? Is it disconcerting or motion sickness inducing to experience motion without some form of rest frame?

• The feasibility of reducing vestibular sensitivity by employing techniques such as GVS (Cevette et al. 2012) or tDCS (Arshad et al. 2015).

MR headsets offer new possibilities for manipulating our perception of motion, capable of presenting content anywhere within the vehicle interior, and may assist in both understanding, and resolving, motion sickness. However, until motion can be adequately integrated into the MR rendering, or the vestibular system can be appropriately stimulated or suppressed, MR headsets will be likely to contribute to increased levels of motion sickness in the passenger use case.

Impediments to acceptance and adoption

At this point, we assume that the MR headset can effectively discriminate between user and vehicle motion, has been certified to be safe to use in the event of a crash event and provides effective measures for delaying or preventing the onset of motion sickness. As such, the headset is comparable in terms of physical safety to usage in non-passenger contexts, and functions correctly in-motion. The challenge now is to provide acceptable and compelling MR experiences that would justify their adoption and usage in-motion. Whilst the scope of this challenge is near-limitless, we focus on four challenges that we consider most pressing: understanding, and designing for, acceptable use in shared transit; supporting interaction in constrained, confined spaces; supporting bidirectional social at-a-distance experiences (e.g. holoportation-style telepresence (Orts-Escolano et al. 2016)); and exploiting vehicle motion and context for entertainment.

Challenge 4: acceptable use in shared transit

Most affected Transportation shared with members of public (e.g. planes, trains, carpooling).

Our use of MR headsets in shared transit is challenged by a multitude of factors: the social and cultural norms regarding our fellow (known and unknown) passengers, the context of our travel environment and the practical challenges introduced by occluding reality (particularly focussing on VR headset usage). These factors are ever-evolving, with subsequent exposure and adoption of MR technology in other contexts likely to influence passenger acceptance and adoption over time. Regarding social acceptability, consider a VR game that uses grabbing gestures in mid-air. Whilst likely acceptable at home or in private, on a public bus such behaviour could be extremely unacceptable and even disruptive to other passengers. When interacting with technology that is highly visible to others, such as a headset occluding the face, the sustained spectatorship of other passengers creates a potentially uncomfortable situation for both users and passengers alike (Wiliamson et al. 2011).

Our travel context in particular would appear likely to affect our willingness to wear an MR headset or engage in an MR experience. For example, travelling with intimacy groups such as friends or family would likely bring with it social pressures to converse or engage in shared activities to mutually pass the time. Conversely, passengers might exhibit very different attitudes towards MR headset use when sharing a flight with collocated strangers (where indeed the VR headset might even be provided by the airline (Air France 2017; Qantas 2015)), versus during a daily commute when carpooling or on the train, versus a late-night bus journey. There are a multitude of contextual factors that could contribute to varying attitudes towards usage here, from the nature of the physical environment and proximity/relationship to other passengers, the duration of travel, perceived personal safety within the mode of transport, etc. And there will invariably be cultural effects, with it repeatedly being noted that different cultures often have varying different attitudes and expectations when it comes to socializing with other passengers in-transit (Baseel 2014; Studarus 2018; Smith 2016). Will the person you are seated next to on your long-haul flight resent your immediate escape into the solitude and isolation of an immersive VR experience?

Occluding reality whilst in transit could even result in a variety of issues that could make headset use unpalatable or even unsafe. Loss of awareness of other passengers means that VR/AR users may accidentally disrupt others or physically invade their space (Williamson et al. 2019). Fellow passengers may not know if they are visible to a headset user and be unsure how or if they can interact with that person (e.g. needing to squeeze past them to stand up). Emergent situations more generally—turbulence on a plane, the entry of drunk passengers to the bus—might vary willingness to engage in MR activities, if the user is even aware that such situations are occurring. As a blogger noted of their VR in-flight experience:

Once it becomes clear that you can see someone through the hardware, even though they can’t see your eyes, people don’t seem to know how to react. I turned on the camera once or twice just to look around, and a few people were openly gawking at me...It also felt way too strange to play any game that forces you to look around in an active way. It seemed almost weird to be sitting in tight space, whipping my head around to look at things only I could see. (Kuchera 2015)

Interaction with other passengers and staff is often unavoidable, for example to ask for directions or to move out of the way. In this anecdotal experience the flight attendant reacted to the VR user by “pass[ing] by without asking if I wanted anything”, given there was no obvious/acceptable way of interrupting the VR user. Safety becomes an issue if headset wearing passengers are unaware of safety announcements and can’t react quickly to dangerous situations. There are also practical reasons to need awareness of immediate surroundings, for example to protect your belongings or to know when to get off of a bus. The fact that users would actively chose to occlude reality and accept illusions may also be unacceptable (Frangakis et al. 2014), leading to tensions between passengers.

Resolving this challenge

Aspects of this challenge may be resolved in time, given changing attitudes and exposure to MR headsets. However, research can perhaps make it easier for users to conceive of using these headsets in shared spaces. For example, social acceptability can be broached through more discreet, wearable and even fashionable form factors (e.g. the Bose Frames (Bose 2019)), and the design of interactions that are equally nondescript to carry out (see Challenge 5). For example, a reliance on gaze-based selection might not be appropriate if it gives rise to the appearance that the user is staring at another passenger—perhaps requiring a different input modality better suited to the plane environment.

Social and cultural norms will inevitably evolve, and we would expect that new norms would arise. But tensions here could be eased, particularly for VR usage, if we could appropriately tackle the issue of occlusion, and awareness, of reality. The subsequent integration of cameras into VR headsets, particularly aiming at inside-out tracking, would appear likely to lead to headsets that can provide mediated awareness of the surrounding sights, sounds and emergent situations of reality, potentially improving social acceptability. For example, co-located people can be visualized in a virtual scene using depth sensing for a mixed reality experience (McGill et al. 2015). Cameras incorporated into headsets can also be used for mixed reality, for example the “Pass Through” views on the Oculus Quest or Gear VR which use a front-facing camera to provide a view to the real world in virtual reality. Some travel contexts may be better suited for early adoption of VR/AR, for example air travel (Williamson et al. 2019; Qantas 2015), where passengers must spend extended periods of time in an enclosed and monitored space where other passengers have been previously security screened.

Of particular note are solutions initially proposed by Williamson et al. for in-flight VR (Williamson et al. 2019). They identified key mechanisms by which the acceptability of in-flight VR usage could be improved, by both facilitating easier transitions between virtual and physical environments by utilizing mixed reality and supporting interruption from co-located “outsiders” such as other passengers or staff. In effect, supporting awareness of the real world and providing mechanisms by which the real world can acceptably encroach upon the VR experience would appear key to providing VR passenger experiences that could see future adoption. Further research will be required to understand how acceptability varies by passenger context and what more the headset can do to make experiences acceptable using the available sensing and peripherals of consumer MR headsets, with significant open research questions regarding:

• Providing awareness of proximate persons How should MR headset wearers be informed of the actions, attention and proximity of other passengers or staff?

• Providing external awareness More broadly, how can passengers be kept aware of external events occurring within the vehicle in ways that do not necessarily break presence/immersion, e.g. if announcements are being made on the train or plane?

• Facilitating interaction between passengers and proximate persons How can other passengers or staff gain the attention of, or communicate with, the headset wearer? How should necessary interruptions be facilitated?

Challenge 5: interaction in constrained spaces

Most affected Transportation with restricted seating in close proximity to others (e.g. economy airline seating).

It can reasonably be expected that MR users in homes and offices will be accustomed to rich support for interacting with virtual content. Currently, it is standard for VR headsets to support either on-headset interactions using buttons or touch-sensitive surfaces, or hand-based interactions using controller peripherals, providing haptic feedback as well as capacitive touch input, and further work is ongoing to incorporate necessary elements of reality into VR experiences, e.g. physical keyboards (McGill et al. 2015; Boland and McGill 2015; VIVE 2018). Conversely, AR headsets such as Hololens have demonstrated peripheral-free, touchless interactions using hand tracking technology. However, existing interaction paradigms for VR and AR headsets do not take into account:

• The physical constraints of a seated MR passenger

• The capabilities and affordances of a given instrumented, connected, interactive vehicle environment (e.g. a car dashboard or plane cabin with seatback display)

With reference to the constraints, the most obvious is that of the physical seating. Regardless of if the user sees it as socially acceptable to perform body-based gestures (Rico and Brewster 2010) or use tracked hand-held peripherals or controllers, the physical environment, seat belts and the proximity of those seated nearby would likely dictate that more discreet gestures or interactions be performed. Regarding existing capabilities and affordances of a given passenger vehicle, the way drivers and passengers use in-vehicle infotainment systems is ever-changing. For example, physical buttons and dials on automotive centre consoles have been substantially reduced with the introduction of touchscreens and touch-sensitive surfaces, already a common sight in long-haul plane journeys. In some cases, touchscreens have replaced all buttons, dials and switches on the centre stack, e.g. the interior of a Tesla Model 3⁶.

Resolving this challenge

Research will be required to explore the suitability of existing MR interactions transposed to constrained, in-motion contexts (Marshall et al. 2016). For example, hand-based gesturing may be impaired by the physical constraints of the environment (e.g. available space, restraint, motion), social acceptability and limitations regarding headset-based sensing. More unobtrusive, eyes-free interactions could bridge this gap, for example the NotifyEye eyes-free rub pad (Lucero and Vetek 2014). Supporting such interactions could require integration of sensing (e.g. Leap Motion (Toppan and Chiesa 2015), Soli (Wang et al. 2016)) and feedback (e.g. Ultrahaptics (Toppan and Chiesa 2015)) into the vehicle. These constraints will also impact how MR (with an emphasis on VR) content is viewed, given that physical limitations regarding neck and head movement would necessitate that either the content be restricted to a narrower field of view than full 360$^{\circ }$ experiences, or some means of scrolling/changing orientation be provided (e.g. rotational gain (Hong and Kim 2016)), to prevent users from feeling frustrated regarding being unable to fully attend to entirety of the virtual 360$^{\circ }$ space.

Where available, touchscreens and touch-sensitive surfaces on centre consoles could provide an additional, richer input modality for users during MR interactions, and give designers the opportunity to develop new input techniques for passengers that are perhaps suitable for MR experiences in the future. However, this will require further advancement regarding how we incorporate necessary elements of reality into MR (and particularly VR) experiences (McGill et al. 2015), and necessitates the interactive environment of the vehicle be tailored or made accessible to the MR headset. Related work has investigated how well drivers and passengers point (Ahmad et al. 2015) and perform common gestures such as swiping (Burnett et al. 2013) on touchscreens. The use of pressure input is becoming more popular with touchscreen smartphones, so researchers have also begun to explore in-car touchscreens and centre console surfaces with force-sensing capabilities to look for alternative input modalities that could be more effective and safer to use in vehicles (Ng and Brewster 2016; Ng et al. 2017). New technologies such as printing sensors and actuators on surfaces such as the dashboard and centre console could provide users with multiple touch surfaces with haptic feedback to interact with AR and VR applications (Frisson et al. 2017).

We suggest there are open research questions regarding firstly how to design new MR peripherals suited to use in constrained spaces, from new positionally tracked peripherals that can function in-motion, to appropriating existing peripherals such as smart watches/phones/rings. And secondly, how to bring tangibility to virtual displays and UI interactions. For example, direct hand-based mid-air interactions with virtual displays or UI elements could be made tangible by appropriating existing surfaces (including existing interactive elements such as touchscreens) and appropriating existing and new feedback modalities (e.g. mid-air ultrasonic haptics has been repeatedly suggested for non-MR passenger use). This latter point in particular is one with much wider applicability to VR/AR usage across a variety of contexts. However, the passenger MR use case may suggest that appropriating existing surfaces is of particular worth to explore, given the existing physical surfaces (e.g. seatbacks, tray tables, doors, arm rests, etc.) typically within reach.

Challenge 6: supporting shared experiences

Most affected Transportation that features a degree of privacy to allow for speech (e.g. autonomous cars)

Social VR has been suggested to be a significant future driver of adoption of MR headsets, precipitating events such as Facebook’s $3billion purchase of Oculus in 2014 [30]. MR headsets have the capability to change how we communicate at-a-distance. Prior to such headsets, communication was limited to voice and video. However, MR headsets have the capability to render virtual content with depth and at a real-life scale and thus support embodied telepresence, where those the user is communicating with at-a-distance are seen to be sharing the same virtual (e.g. McGill et al. 2016; VRChat 2018) or physical (e.g. Orts-Escolano et al. 2016; Fanello et al. 2016) space. This application could be suited to transportation where social acceptability is less of a concern, for example use in private autonomous cars.

Resolving this challenge

Firstly, there is the question of how the passenger should be captured and portrayed at-a-distance. Currently, shared at-a-distance VR applications typically display an avatar conveying head movements and voice [30]. However, advances in depth camera technology can allow for body tracking and embodied telepresence (e.g. McGill et al. 2016; Orts-Escolano et al. 2016; Fanello et al. 2016, see Fig. 5) where the remote user is captured and rendered in 3D. To perform said capture of passengers in-car would require the integration of depth camera technology, thus it is likely that more cost-effective approaches will need to be investigated, e.g. portraying avatars on the basis only of what sensing is available on the headset (Hoffman 2016). Secondly, it is also unclear how best to render others at-a-distance in such environments, especially for AR headsets that have to in some way incorporate reality in the presentation. Telepresence literature has often concentrated on room scale spaces, where a mapping between two disparate physical places could be constructed, essentially allowing direct interaction with virtual avatars (Pots 2016; Orts-Escolano et al. 2016; McGill et al. 2016; Fanello et al. 2016). However, when using an AR headset in the constrained environment of a car or plane, such interactions are no longer feasible, as the existing physical space needs to be taken into consideration in the presentation. In such cases, should those at-a-distance be incorporated into the car or plane as other virtual passengers taking up empty seats, or rendered world-in-miniature for viewing by all passengers, etc.? It may even be the case that existing video communications enacted over AR (Kun et al. 2017) are the most appropriate for such constrained spaces.

Fig. 5

Examples of embodied virtual social experiences. Left: VRChat, a VR social platform with voice chat and customized avatars for lower fidelity, but more broadly accessible, embodied telepresence (VRChat 2018). Middle: a VR-based mixed reality experience where users could see each other captured in real-time and view a synchronized $360^{\circ }$ experience (McGill et al. 2016); Right: an AR-based mixed reality experience, “Holoportation”, where users could see each other captured in real time (Fanello et al. 2016). Each brings with it different logistical challenges in terms of capture of users and presentation whilst in-transit

We suggest there are open research questions regarding:

• How to capture and convey passengers? Different forms of transportation will likely be suited towards different sensing technologies for capture.

• How to render at-a-distance participants? This is of a particular challenge for AR headsets, as any telepresent portrayal would be constrained by the physical environment of the passenger.

Fundamentally, mobile in transit MR should strive to maintain parity with non-mobile MR, given the predicted importance of social interaction at-a-distance for MR headsets. To do so will require an understanding as to how best to capture and convey telepresent users, such that as many of the benefits of face-to-face interactions as possible are retained regardless of mode of transport.

Challenge 7: exploiting vehicle motion and context for presence and immersive experiences

Most affected All forms of transportation. The motion, location and context of the vehicle itself can be instrumented and utilized to potentially create more novel, engaging, immersive experiences. This point notably extends the discussion on motion sickness: where previously motion was being integrated to alleviate sensory mismatch for generic MR content, here the intention is to both alleviate the sensory mismatch whilst more deeply integrating the experience of motion into the virtual experience, taking advantage of sensory alignment (Marshall et al. 2019). The outcome of this could, for example, be turning your ordinary car journey into an exciting 100 KPH space battle, providing a more affective experience by using the vehicle as a motion platform.

Kodama et al. (2017) categorized the use of cars as motion platforms for VR content as being either (a) an active virtual drive system, meaning that the VR user controlled the car, with the experience in VR reflecting that of reality (effectively substitutional reality (Simeone et al. 2015)); (b) a passive virtual drive system, meaning that the motion of the car was integrated into the VR experience, with content limited by the driving route, but the VR user exhibited no control over this (e.g. in autonomous cars/the passenger use case); (c) A content player system, whereby the motion of the car was synchronized with the VR environment.

They examined a content player system, where users had limited control over the car (controlling acceleration over a 7m track) and the content was in synchrony with the motion along one axis (acceleration forward). The VR content was effectively a rollercoaster, and it was noted that, due to the congruent visual and physical motion cues regarding the forward motion, the additional visual cues of the rollercoaster going downhill induced a significant sensation of falling, despite there being no physical vertical motion experienced. This concept of having VR users control their vehicle was extended by Goedicke et al. They adopted a “fused reality” approach where passenger driving actions could be simulated by a real driver as a WOZ prototype (Goedicke et al. 2018), as a means to creating VR driving simulations which retained the “immediacy and rich sensations of on-road driving”. These papers help to illustrate the breadth of ways in which the relationship between the MR-wearing passenger and vehicle can be exploited.

Resolving this challenge

In reviewing vehicular MR papers, we identified eight factors that could be described as using the vehicle to positively affect immersion:

• Control over motion Ranging from active (full MR user control) to passive (no control, MR user is passenger). As noted by Kodama et al. (2017), this is perhaps the most problematic factor, as there are clear safety concerns regarding how much control the user is given over the experience. However, envisioning a fully autonomous car where the virtual experience or user has some measure of control over acceleration (e.g. varying between 80km/h and 120 km/h on a motorway) could be feasible in certain circumstances (e.g. a clear road), as demonstrated by Goedicke et al. (2018).

• Foreknowledge/synchrony Both of the route and the impending vehicle actions, building upon the previously discussed benefits of anticipatory movements and actions (Sawabe et al. 2017) towards potentially increased presence, e.g. seeing and feeling your virtual experience follow the same path (Goedicke et al. 2018). Used particularly by Paredes et al. to create a calming, mindful VR experience in-car (Paredes et al. 2018).

• Context Knowledge/sensing of the specific real-world location/context, e.g. rendering overlays on other cars in AR experiences (Dent 2017), integrating elements of the real world, such as landmarks into the passenger experience (Baldwin et al. 2017), or reacting to detected events (Rober et al. 2018) such as a red light being conveyed as a temporary virtual wall.

• Motion profile Different vehicles can have very different motion profiles. Compare a cross-country train with long periods of relatively constant velocity followed by long, steady changes in acceleration, versus cruise liners or yachts where the acceleration profiles will be modest; however, oscillations and orientation changes are much more prevalent (and at very different frequencies).

• Conveyance of motion Movements could be conveyed in terms of changes in acceleration, or absolute velocity. Each would have implications for the kind of MR experience being presented, e.g. on the rails virtual journeys versus seemingly stationary experiences with additional visual cues of motion (e.g. particles moving around the user, displays moving back and forth based on accelerations (Hanau and Popescu 2017), etc.).

• Magnitude of motion The transfer function between real-world motion and MR-rendered motion has been shown to be able to be varied significantly in roomscale VR (Wilson et al. 2018a) without impacting simulator sickness. Translational and rotational gain could potentially be manipulated to enable more exciting (e.g. conveying 30km/h in reality as 100km/h in VR, or conveying a modest acceleration as a significant one) or calming (conveying accelerations as if gentle movements of a rocking chair) virtual motion experiences.

• Environmental control Consider concept cars where temperature, air flow and even odour (Dent 2017) can be controlled by the virtual experience to increase the user’s sense of presence in a virtual experience.

• Anchors/rest frames Rest frames have been noted to be helpful in preventing motion sickness onset (Hock et al. 2017; LaViola 2000), and such visual anchors could be exploited to convey very different environments, e.g. the cockpit of a spaceship, exploiting substitutional reality to render virtual elements that are physically congruent with the vehicle interior (Simeone et al. 2015).

There have been few other concrete examples of exploiting motion for immersion thus far. With respect to VR headsets, Soyka et al. simulated a flight experience on a virtual magic carpet ride, with the intention that airline passengers would experience their journey across a virtual landscape with unrestricted views (Soyka et al. 2015), whilst Hock et al. presented the movements of a car in a virtual cockpit of a helicopter flying over a pre-generated virtual landscape based on a predetermined route for the car (Hock et al. 2017) (Fig. 6). In effect, the car journey was gamified, turned into a first-person virtual helicopter shooter, with passengers flying around a new and different landscape during their journey. Hock et al. found that the kinaesthetic forces perceived by users increased enjoyment and immersion, whilst reducing simulator sickness. Similarly, there exists a number of consumer rollercoaster rides where the virtual experience is tightly linked to the physical motion perceived, to varying degrees of success (Ion 2016). This congruence of visual and physical perception of motion delays or prevents the onset of motion sickness. Moreover, Hock et al. noted that the virtual portrayal of car motion resulted in some participants completely losing awareness of where they were, as well as distorting their awareness of the passage of time. This suggests that occluding reality can in part aid the passing of time in transit, a notable potential benefit for long-haul flights for example. Commercially, Apple have submitted a patent regarding congruent in-car experiences (Rober et al. 2018), whilst Renault and Ubisoft have demonstrated a VR concept car (“Symbioz”) which exploited foreknowledge and context to render motion:

A minute ago I was on a real road, but now I’m rolling down a fake forested highway in a simulation created by Ubisoft. Meanwhile, Renault’s Level 4 autonomous system has taken the piloting chores...It’s a bizarre experience, but I don’t feel sick, because the Symbioz is transmitting real road motion to the headset...I even see simulated versions of the cars and trucks on the road fed in by LiDAR and other sensors. (Dent 2017)

With respect to AR, whilst there are a number of discussions regarding AR use in-car for aiding driving and navigation, there are few examples of AR headset usage by passengers currently. However, discussions of AR windshields and windows (Rao et al. 2014a; Häkkilä et al. 2014; Haeuslschmid et al. 2016; Rao et al. 2014b) hint at potential applications, for example augmented annotations of locations and landmarks (Baldwin et al. 2017; Large et al. 2017) and identifying points of interest for tourism.

Fig. 6

From left to right: (1) COMS-VR system, where VR users had control over vehicle acceleration (Kodama et al. 2017; 2) VR-OOM driving simulator in-car (Goedicke et al. 2018; 3) Turbulence from a simulated flight experienced on a virtual magic carpet ride, from the VR HYPERSPACE project (Soyka et al. 2015; 4) A virtual helicopter game where movements in-game match the forces experienced during a real-world car journey, from the CarVR project (Hock et al. 2017)

Broadly, what these prototypes demonstrate is that the motion and location of a vehicle can be integrated into MR experiences in a variety of potentially engaging and affective ways. Virtual gamification of vehicle motion and using AR as a virtual tour guide are two examples that have the potential to fuel adoption of MR headset usage in vehicles whilst retaining a direct link between what is visually and physically perceived, the most immediately accessible use case of MR headsets in transit when considering motion sickness. Moreover, such experiences are unique to transit because they require some congruent element of reality to be incorporated, providing additional value to MR headset use beyond standard entertainment and productivity. However, a breadth of further research will be required to understand the design parameters for each of the factors identified.

Discussion

MR headsets have the potential to significantly improve the passenger experience. However, there are a number of impediments to the adoption and usage of MR headsets when in transit. We have discussed three functional impediments, identified through a combination of our initial research into this domain and a review of the emerging literature, being challenges that would prevent or limit normal usage of MR headsets in transit:

• 1. Maintaining a forward bearing in a moving world

• 2. Crash safety

• 3. Motion sickness resultant from discongruent or conflicting visual and physical motion cues

These challenges need to be addressed to provide a foundation upon which VR and AR experiences can be built that function correctly, have access to the necessary vehicle telemetry and do not endanger the passenger. We suggest that these challenges will require cooperation between academia and industrial partners across the domains of transportation, augmented reality and virtual reality.

We have also discussed four significant impediments to the adoption and acceptance of MR headsets. By this, we mean “softer” problems that need to be understood to provide compelling experiences that would justify the adoption of MR headsets in transit, being:

• 4. Enabling socially acceptable use in shared transit

• 5. Facilitating interaction in constrained spaces

• 6. Supporting social at-a-distance mixed reality experiences

• 7. Exploiting vehicle motion/context for more engaging and affective passenger experiences

These challenges are not critical to the basic function of headsets in-motion. However, they should be investigated to arrive at MR headsets that can feasibly support adoption in a wide range of use cases and provide parity with non-mobile usage. Indeed, parity is perhaps the most important point: adoption of MR headsets in the home and office will lead to user expectations that such usage will be able to continue unabated in transit. Accordingly, users should be provided with as close to the same capabilities when in transit as possible, such that they might perform the same actions, view the same content and communicate in the same way at-a-distance.

However, it is important to note that the need to address many of these challenges is predicated on MR headsets falling into favour with consumers. VR headset adoption has recently stumbled, with the suggestion being that the cost, limitations in terms of fidelity, and a lack of compelling experiences and use cases have led to some consumer apathy, with suggestions that VR is in the “disappointment phase” of the consumer hype cycle (Skarredghost 2018). Microsoft, for example, suggested that immersive VR headsets “did not meet, in general, the high expectations that were set for them” (Feltham 2019). AR headset adoption is not yet even a feasible possibility, with no compelling consumer-level headsets currently available at the time of writing, although this will inevitably change. Over a long enough time line, it would seem reasonable to suggest that passenger adoption of VR headsets would at least be likely in long-duration journeys in comparatively uncomfortable environments, e.g. economy seating in long-haul flights. The likelihood of AR headset adoption would appear stronger over the coming years, particularly if AR headsets reach a point where they become standard, everyday consumer devices. For both AR and VR, compelling use cases such as productivity and entertainment will help to drive headset adoption only once headsets reach an inflection point across cost, fidelity, sensing capability, interaction, social acceptability and fashion, amongst others. If such a point is reached, it could be expected that demand for passenger usage would soon follow.

As recognized by both car manufacturers such as Dent (2017), technology companies such as Apple (Rober et al. 2018) and commercial airlines (Air France 2017; Qantas 2015), the reward for facilitating such usage could be significant. MR headsets have the potential to provide new and varied ways by which travellers can make use of their time in transit, and could provide an additional motivator towards the adoption of transportation such as autonomous cars. Moreover, their adoption could provide a new market of MR headset users and bring with it significant economic benefits for society. However, arriving at this point will require a multidisciplinary research effort involving members of the HCI community, vehicle and headset manufacturers, and others in fields such as systems engineering, sensing and health. This paper serves to outline the challenges involved in facilitating MR headset usage in transit, with the aim of realizing their potential to transform the passenger experience.

Conclusions

This paper has discussed key challenges in supporting passenger use of augmented and virtual reality headsets in transit. MR headsets have a number of advantages over current seatback and mobile displays, e.g. in terms of privacy, field of view and immersion. However, passengers are not yet in a position to fully realize the benefits of using headsets in transit as there are significant barriers to their usage. These barriers range from impediments that would entirely prohibit their usage (e.g. unknown crashworthiness, motion sickness) to those that might prohibit their adoption (e.g. social acceptability and a lack of parity between everyday usage and the capabilities of headsets in transit). We identified and discussed seven key challenges preventing safe, fully functional usage of MR headsets in transit, with the aim of facilitating more research into this nascent application area, assisting passengers to make better use of their time by making travel more productive and enjoyable.