1 Introduction

In recent years the continuing development of fully autonomous vehicles (AV’s), also referred to as driverless and self-driving cars, suggests that it will not be long before these vehicles are seen on public roads. Now, with advances in technology and the search for the next ‘big thing’, many companies are involved in AV development, either from scratch, or by implementing an automated driving system (ADS) and functionality into existing models such as Nissan, Tesla, Audi and Volkswagen. Other companies focus on the development of aspects of hardware and software required to support AV’s.

The market for vehicles with some level of autonomy is predicted to undergo significant growth over the next 2–3 decades, supported by continuing technological development and falling production costs (Lavasani, et al., 2016). One analysis suggested the market share would reach 8% by 2025 and 50% by 2035 (J’son & Partners Management Consulting, 2017) at varying levels of driving automation. As the AV development continues and production costs fall, the sale price of AV’s is expected to fall increasing purchasing (Xie & Liu, 2022). With these predictions in market growth, accompanying changes are being made to AV regulations and laws around the world. For example, an update to the German Road Traffic Act was recently announced, allowing SAE Level 4 vehicles on roads (Kettwich et al. 2022) and in France fully autonomous vehicles can now be allowed on roads for testing (Connected Automated Driving.eu, 2021). As part of the Queens Speech in early May 2022, attention was brought to the planned changes to the Highway Code in the UK to help the integration of AV’s onto UK roads including clarifying at what point a driver must be ready to resume control (Centre for Connected and Automated Vehicles 2022). Requirements for trials on UK roads require a suitably trained and licensed safety driver, however it is now possible that this safety driver may be remotely located as long as they are able to exercise suitable control over the vehicle (Centre for Connected and Automated Vehicles 2022).

The increasing prevalence of AV’s is associated with many potential advantages, for example even at a low market penetration rate of 5%, smoother traffic flow patterns may be achieved, and emissions reduced (Stern et al. 2018; Bagloee et al. 2016; Talebpour and Mahmassani 2016). This is realised through improved ‘string stability’ which occurs when a disturbance is not amplified as it propagates along a vehicle string (Feng et al. 2019; Qin and Orosz 2020). Platooning allows autonomous and connected vehicles to move in the same lane whilst maintaining a consistent distance (Liu et al. 2019) which can be used to allow vehicles to follow more closely reducing congestion, aerodynamic drag and vehicle emissions (Wang et al. 2019; Greenblatt and Shaheen 2015), especially for similar sized vehicles (Alam et al. 2010). The automation of vehicles has the potential to improve mobility for many user groups who do not currently drive, such as those with driving-restricting medical conditions, and those too young to hold a license (Harper et al. 2016). AV’s may have additional advantages such as increased passenger space and privacy compared to conventional taxis (Piao et al. 2016); in fact, vehicles may be completely redesigned to prioritise occupant comfort and accessibility. There is also the potential for increased ride sharing through shared autonomous vehicles (SAV’s). SAV’s combine existing car-share services with AV technology and may make ride sharing more accessible by reducing walking times to vehicles and resolving relocation issues and cost issues of one-way trips (Firnkorn and Müller 2015). It may also result in a reduction in fleet numbers required (Othman 2022; Fagnant and Kockelman 2014) and result in lower emissions, improved traffic conditions and reduced parking requirements (Zhang et al. 2015).

Despite the potential benefits of the introduction of AVs there remain significant challenges. The transitional period, where both conventional and autonomous vehicles coexist, will cause complex issues for traffic modelling and management as human-controlled and autonomously-controlled vehicle behaviours interact (Tettamanti et al. 2016). The introduction of new technology to the public may also cause a variety of Human Factors issues, for example the need for additional human-machine interfaces (HMIs) inside vehicles and changes to the interactions with road users in the wider environment. Displays on the outside of vehicles (external HMI’s) may be required to facilitate communication with other road users in the absence of a human driver (Lee et al. 2021). There will also be a period where AVs are still ‘learning’ and cannot be considered to act perfectly under all circumstances (Fleetwood 2017). Performance issues are likely to emerge in situations that cannot be fully predicted in advance. It is in these scenarios, where ADS fails, is unable to process the scenario or the vehicle user is unable to interact with the vehicle that a remote operator (RO) may be used. In fact, even after this interim period, edge cases will still occur causing unpredictable issues for AVs which are likely to require human intervention (i.e., remote operation) to solve.

2 Argument for remote operation in autonomous vehicles

In instances where the ADS is unable to successfully deal with a road scenario, it is envisioned that an operator who interacts with the vehicle from a remote location may be used to bridge the gap. This remotely located human, or RO, can support autonomous driving by enabling the journey to continue when the AV technology is insufficient for identifying and resolving issues. A recent report from the Law Commission on Remote Driving recognises the potential benefits of enabling an operator to control a vehicle from a remote location (Law Commission of England and Wales & Scottish Law Commission, 2022). Remote operation itself is not a new concept: any operation of a system or machine at a distance, falls under the umbrella of remote (or tele-) operation. Throughout this paper, ‘remote operation’ is used to describe the actions of a remotely located human to control an AV, as ‘teleoperation’ is not defined consistently within the literature as noted in the SAE International (2021) taxonomy and definitions related to ADS in on-road motor vehicles (SAE International 2021a). As remote operation performed beyond-line-of sight is a new concept in road vehicles new challenges are posed to those acting as RO’s. The focus of this work is on remote operation used beyond-line-of sight to support the utilisation and action of high-level AVs without user-accessible controls.

Not all vehicles operate at the same level of automation; SAE International (2021) produced 5 Levels of Automation (LOA), which have been accepted as a standard definition by many through the industry (Inagaki and Sheridan 2019). This includes a base level, 0, where the features are limited to providing warnings and momentary assistance such as automatic emergency braking. Levels 1 to 5 can be generically referred to as a ‘driving automated system’ recognising the hardware and software are capable of performing part or all of the DDT on a sustained basis, but ADS is used to specifically refer to levels 3–5 (SAE International 2021a). As levels 0–3 reference vehicles with either no automation or features where the driver is still considered to be driving or must drive under request, they are outside the scope of this research. Instead, the focus is on Levels 4 and 5 which do not require the user to take over the automated driving features. At level 4 the AV can operate when certain conditions are met. This may be situational, such as a traffic jam chauffeur, or within an operational design domain (ODD) where the vehicle may not have steering wheels or pedals installed and can operate completely autonomously within that specified area or condition. At level 5, the ADS can operate the vehicle everywhere under all conditions. However, there may be cases, especially in early adoption and integration of Level 4 and 5 AV’s, when the vehicle encounters scenarios that its programming is unable to deal with (SAE International 2021b). Due to the likelihood that there will be no way for the vehicle occupants to take control of the vehicle (i.e. due to the absence of input controls), remote operation is posed as a potential solution to dealing with these unpredictable situations. These may include negotiating road blockages and accidents, but will also encompass many more emergency scenarios.

Remote operation is used in many industries, making use of advances in technology such as improved sensors, communication infrastructure and tracking software. However, including remote operation in AV infrastructure necessitates investigation to understand the requirements for humans to be able to work in this position efficiently and effectively. The cognitive requirements of the RO must be considered, such as situational awareness (SA) as the increased automation means the human is further removed from the loop at the point of vehicle interaction or takeover (Onnasch et al. 2014). Ensuring SA is maintained without creating high levels of workload is important to prevent a negative impact on operator performance (Mutzenich et al. 2021a). Other Human Factors issues include RO workstation design (particularly information presentation) and job design (with respect to different roles and relationships between roles). The role of a remote operator will also be subject to real-time changes based on the road situation, which may be impacted by weather conditions, traffic, local events, etc. which vary in predictability (Goodall 2020). Further, RO’s will be subject to many of the legal and regulatory decisions around AV’s, particularly relating to the protection of personal information, security, responsibility, and safety (Skrickij et al. 2020). It will be important to understand the implications of all these factors for the RO role.

Many companies in the AV sector have included remote operation in their operating plans. Although this information is presented more in advertising than as technological information, the descriptions of some of these systems are available. Nissan, for example, has the Seamless Autonomous Mobility (SAM) system, developed with NASA, to use remote human support to help autonomous vehicles make decisions in unpredictable situations which they expect to include obstacles on the road (Nissan Motor Corporation, 2022). Valeo demonstrated in 2019, Drive4U Remote, their remote operation system expected to take over in unprecedented situations such as weather events or a health problem (Valeo 2019) and Zoox offers TeleGuidance where the remote ‘human-in-the-loop’ is able to offer guidance to the vehicle without taking control (Zoox 2022). There is some variation between companies on how this may be utilised, some expect full remote driving, i.e. the RO has full control over the driving task, whereas others expect only an assistance role. However remote operation is conducted, it seems clear that many see it as a solution to the issues occurring in operating AV’s in unpredictable scenarios.

2.1 Defining remote operation in driving

The driving task, as defined in SAE International (2021), is ‘all of the real-time operational and tactical functions required to operate a vehicle in on-road traffic, excluding the strategic functions such as trip scheduling and selection of destinations and waypoints’ (SAE International 2021a). The expanded definition references the three hierarchical levels of control in driving: operation, tactical and strategic (Michon 1985; Matthews et al. 2001) and indicates which elements of the driving tasks meet each level of control. This includes subtasks of lateral and longitudinal control (i.e. operational tasks) and tasks such as object detection, recognition, classification and response preparation leading into object and event response execution, all involving both operational and tactical work. Tactical control of manoeuvre planning and enhancing conspicuity (i.e. by sounding the horn, gesturing or using lights) are also included in the definition of the dynamic driving task (DDT) (SAE International 2021a). All these tasks, and others not listed, may require completion by the RO, depending on the scenario, when the ADS is unable to act as intended to complete the DDT.

An aim of this paper is to define the different roles of an AV remote operator as a starting point for analysis of the information requirements of this job. Currently there is no single widely accepted definition of RO roles and different authors have proposed different approaches. One method is to consider the types of control the operator may have, either direct or indirect. Direct control mimics the manual driving process requiring continuous concentration on the driving task with all elements of the DDT controlled by the RO whereas indirect control requires human intervention and decision making which is then translated into driving action by the vehicle system (Kettwich et al. 2021). Indirect control may preferable as it should be more adaptive to lag issues as the RO gives instruction for small manoeuvres or decision making on a tactical level, avoiding control on an operational level (i.e. deceleration) which is more susceptible to issues with latency and SA. However, in some situations the lack of direct influence and ability to immediately react without waiting for system recommendations means indirect control could be inappropriate and direct control is required. The Law Commission has also given tentative definitions for use during a recent paper, which specifies driving tasks such as steering, braking, accelerating and monitoring the driving environment constitute as remote driving when performed by a remotely located human and simply advising the autonomous system of which route to take is considered as acting as a ‘remote assistant’ (Law Commission, 2022).

An alternative division of RO tasks was developed by Mutzenich et al. (2021b) splitting tasks between the subheadings of Remote Assistance, Remote Management and Remote Control. In Remote Assistance tasks, the RO has no control over driving action and acts in a somewhat customer service-style role, supporting the user of the vehicle, for example by informing the user how long a repair for a flat tyre may take and if a replacement vehicle will be dispatched. Remote Management is used to describe tasks in which the RO has control over fleet operations such as dispatch services and monitoring traffic information. They are also able to authorise road law restrictions if required by the scenario and operate in an advisory role, for example reviewing reasons for a vehicle entering Minimal Risk Condition (MRC) and confirming if safe to proceed or selecting the most appropriate path from available options. An MRC is defined by SAE International (2021a) as ‘A stable, stopped condition to which a user or an ADS may bring a vehicle after performing the DDT fallback in order to reduce the risk of a crash when a given trip cannot or should not be continued.’. This definition assumes the vehicle must come to a stop to be considered in an MRC however it includes the note that a moving condition may also constitute minimal risk, in a scenario where a full stop would increase the risk to occupants and others. MRCs are relevant to the role of a RO because they allow time for the RO to become aware of the incident and build knowledge to allow them to interact with the AV, safely and successfully. In an incident such as a collision it is unlikely an RO could gain understanding and give inputs that would make a meaningful change to the scenario in the initial moments after a collision. Therefore, MRCs can exist to bridge the gap between loss of automated control and the presence of remote operation. The SAE definition is also referenced within other reports, for example it is used to explain terms related to remote operation in a TRL report on the remote operation of connected and automated vehicles (CAVs) (Kalaiyarasan et al. 2021). Remote Control is equivalent to direct control (proposed by Kettwich et al. 2021), where the RO assumes sole responsibility for the DDT from a location remote to the vehicle (Mutzenich et al. 2021b). In an informal document, the United Nations Economic Commission for Europe (2020) proposed three categories of remote support and control: Remote Assistance, Remote Management and Remote Control (UNECE 2020). These headings for the roles are identical to those produced by Mutzenich et al. (2021b) and there are some similarities between them, unsurprisingly, as the paper makes several references to the UNECE report. In the latter definition, Remote Assistance covers communication and support with the user and working with external agents such as towing companies, compared to the Mutzenich, et al. work, which additionally specifies Remote Assistance as including monitoring passenger behavior. This is addressed later in the UNECE report in a section focusing on supporting the needs of passengers without drivers in the vehicles. However, these kinds of support and experiential tasks are not assigned to a particular RO type by the UNECE. The report suggests monitoring of the on-board vehicle environment may also occur within Remote Assistance, although it does not address monitoring of the external, or wider, environment. Remote Management is suggested by the UNECE as analogous to air traffic control and covers a range of tasks including approving path deviation, some path guidance and assisting in hazard detection. Remote Control as defined by the UNECE report is similar to the above definition of Remote Driving developed by the SAE (2021a), with emphasis on temporary or full control of the DTT at both low and high speed but also includes limited path guidance, which is not addressed by Mutzenich, et al.

SAE International (2021a) has also proposed definitions for different kinds of remote operation, with all involvement split into either Remote Assistance or Remote Driving. This definition of Remote Assistance is more akin to the definition of Remote Management produced by Mutzenich et al. (2021b) and is defined as ‘Event-driven provision, by a remotely located human, of information or advice to an ADS-equipped vehicle in driverless operation in order to facilitate trip continuation when the ADS encounters a situation it cannot manage’. Importantly, this definition does not include instructions relating to destination or dispatch functions, even if these are performed by the same person. SAE’s Remote Driving is defined as ‘Real-time performance of part or all of the DDT and/or DDT fallback (including, real-time braking, steering, acceleration, and transmission shifting), by a remote driver’. This is equivalent to the definitions of direct control (Kettwich et al. 2021) and remote control (Mutzenich et al. 2021b; UNECE 2020) previously described.

Although there are some similarities among the various classifications that have been proposed, particularly between those proposed by Mutzenich et al. (2021b) and the UNECE (2020), the current literature still paints a somewhat confusing picture of the work that ROs may be required to do and the responsibilities they may have. Some important steps of human information processing are also omitted in the existing definitions. The implication from many of the works referenced above is of a remote operator working in their role in isolation on a 1–1 basis with the affected AV or AV occupant, and that a remote operator is trained specifically in that role. However, separating ROs into distinct roles and excluding some monitoring elements from these roles, may prevent ROs gaining an awareness of the wider situation, potentially making them less able to deal with unfolding events within the road environment and to collaborate effectively as part of a large, distributed ‘team’. In the current work, remote monitoring is included as an additional role to reflect the importance of creating distributed situational awareness across all agents involved in remote operation, including vehicle occupants. Consequently, to consolidate and extend understanding of remote operation, four levels of RO role are proposed: remote monitoring (RMo), remote assistance (RA), remote management (RMa) and remote driving (RD). A further assumption is that these are not isolated roles and may be better described as competencies which ROs can be trained to achieve and may enact at appropriate times during a single shift and in collaboration with other ROs as required by the situation. This 4-role definition is an extension to the three roles defined by Mutzenich et al. and the UNECE which addresses the requirement for monitoring and anticipating as essential functions in human information processing, following the work on vehicle automation by Banks et al. (2014). Banks extended the information processing taxonomies proposed by Endsley and Kaber (1999) and Parasuraman et al. (2000) to increase applicability to the driving task, highlighting the importance of anticipation and recognition. In the four-level definition proposed in the current work, RMo covers the requirement for information to be gathered from a variety of sources relating to technical and human agents in the system to inform decision making, which then supports the other levels of RA, RMa and RD. It may support the real time decisions of the RO and be used to proactively identify decision points and relevant information. This stage of information processing is missing from the definitions proposed by other authors and the new four-level approach presents a more comprehensive classification of remote operation from a human-centred perspective.

Definitions for each level of remote operation as identified in this study are as follows:

Remote Monitoring (RMo) – Remote observation of autonomous vehicle, user state and environment factors to gather information allowing the prediction and identification of issues to inform decision making.

Remote Assistance (RA) – Remote provision of assistance and/or information to the vehicle user or external agents in close proximity to the autonomous vehicle such as emergency services or vehicle recovery.

Remote Management (RMa) – Remote provision of instructions to an autonomous vehicle to initiate system actions when the vehicle systems are unable to proceed individually, on a 1–1, convoy or fleet basis. Instruction may also involve fleet management and dispatch.

Remote Driving (RD) - Remote control over the dynamic driving task (DDT) of a single autonomous vehicle for a limited time period, where RA, RMa and RD have proven insufficient to resolve issues of vehicle function.

In this study, these four RO roles are mapped onto operator sequence diagrams for a variety of level 4 and 5 AV scenarios. This process was used to demonstrate the suitability of the four roles in describing the different tasks of a RO and is explained in the following sections.

3 Scenario identification

Identification of potential scenarios where the involvement of ROs occurs is important to develop understanding of the roles and subsequent information requirements of the ROs. This began by identifying possible scenarios of AV operation within literature and by analysing disengagement reports of AVs currently undergoing testing. Some scenarios such as unexpected obstructions, unfavourable weather conditions and AV user health problems have already been predicted by companies and included in their reasoning for developing their own remote operation systems. As there has been research into the presence of level 4 and 5 vehicles on the road, some scenarios have been proposed in this research as reasonable scenarios where the AV system may require human intervention. Some suggestions for potential scenarios include damage or failure of the vehicle software and hardware such as mechanical failure (Kettwich et al. 2021) or sensor perception failure (Mutzenich et al. 2021b) preventing operation. Other scenarios may occur due to external environment factors such as the AV being required to follow a path indicated by an external person (i.e. path planning) (Kettwich & Dreβler, 2020) or construction work creating mixed road signals (Rosenzweig 2020). A TRL report produced multiple scenario proposals for remotely operated vehicles rated by scales of risk for different applications of connected autonomous vehicles (CAVs), not just for road vehicles. Among these scenarios were rerouting around an obstacle that required violation of road rules and calling for roadside assistance for the vehicle and users (Kalaiyarasan et al. 2021).

Other scenarios can be identified by projecting the events seen in scenarios encountered in AV disengagement records from testing in California (DMV State of California 2022). Testing records covering 2019 and 2020 were analysed as part of this study. These records covered reasons for disengagement from autonomous activity from vehicles which were part of the Autonomous Vehicle Tester (AVT) programme. Testing without a safety driver was not allowed when these records were taken; in California, driverless testing permits have only been issued since the 19th of November 2021 to seven companies including Cruise LLC and Waymo LLC. The AVT Records cover scenarios including technology failure and the AV encountering situations that required a test driver to take control for safe operation. Although some resulted from the level of automation being tested, and reflected the prototype nature of some companies’ technologies, other reasons for disengagement included those that may also affect Level 4 and 5 vehicles, such as sensor perception failures due to glare, disengagement resulting from system integrity checks, and unsuitable weather conditions. The open format of submission to the DMV means that there is little standardisation to the reporting so it is difficult to produce in depth analysis, however the results can still be used to suggest potential reasons for remote operator involvement and these can be cross-referenced with those suggested in research literature.

Kettwich et al. (2022) compiled a taxonomy of scenarios for remote operation sourced from several AV contexts including observations of control centre staff working in public transport and follow-up interviews, video analysis of road events, and interviews with existing safety operators of AV’s (Kettwich et al. 2022). This taxonomy is split into ‘use case clusters’ each with several ‘sub-use’ cases. These use case clusters can be split into two sections, the first is central interactions compromising RO-passenger, RO-AV, AV-infrastructure, RO-infrastructure and AV-other road users. The second section includes the use-case clusters of RO-State (i.e., RO illness or distraction), contextual factors and technical malfunctions. This taxonomy of scenarios is not considered by the authors to be exhaustive, and it is noted in the discussion of the paper that the levels of depth and detail are varying, affecting the balance of use cases across clusters. Further, relating to the work presented as part of the current study, the taxonomy only differentiates between remote driving and remote assistance as options for the RO, and in some cases does not consider involvement of the RO at all, meaning not all scenarios presented in the taxonomy may be appropriate (Kettwich et al. 2022). Nevertheless, the Kettwich et al. taxonomy was useful to cross reference all of the scenarios identified in this work (through other academic literature and the DMV reports).

Based on the review of literature and comparison with the work of Kettwich et al. (2022), a number of potential scenarios for remote operation of level 4 and 5 Avs were created. These scenarios were identified to represent a range of tasks and activities involving ROs and additional agents in the system, including AV users and external agents (e.g., recovery operators, emergency services personnel). This breadth of coverage RO tasks achieved by these scenarios is thought to be of use in specifying job roles, both in current work and in future studies. A set of twenty-one scenarios were generated and eight of these scenarios are presented in Table 1 to exemplify typical tasks and levels of remote operation (RMo, RA, RMa and RD):

Table 1 RO scenarios and descriptions
Full size table

4 Creation of operator sequence diagrams

Previous work by Banks et al. (2014), used operator sequence diagrams (OSDs) as a representational aid to explore the way that levels of vehicle automation affect system network dynamics. Banks et al., used a single scenario, a pedestrian warning/ detection system, to explore the interactions between several different agents (driver, vehicle, warning/detection system, auditory warning, visual display, and the environment), over a range of automation levels from manual control to full automation. The OSDs were able to produce useful insights into how human-vehicle interactions were influenced by changing levels of automation, including for scenarios at a higher automation level than was currently possible. This suggests that OSDs are also a suitable method for predicting the relationships between agents in a remote operation context, where there is limited information on how the scenario may develop in detail (Banks et al. 2014). More recent papers have also utilised OSDs to model vehicle automation to human driver handover, with validation showing reasonable confidence in the results produced from this method (Stanton et al. 2021).

OSDs are often used to graphically describe the interaction between multiple agents within a system, usually depicting the task process through a series of standardized symbols (Stanton et al. 2013). Each system in the scenario has its own column, or ‘swim lane’, with the activities of each and the interaction between them graphically depicted using standardised symbols (Stanton et al. 2022a). Initially developed to represent complex tasks with multiple agents (Kurke 1961), OSDs can be easily changed to introduce new ‘swim lanes’ to incorporate more functions in the wider system (Harris et al. 2015). Also referred to as operator event sequence diagrams (OESDs) in literature, previous applications of OSDs include illustrating distributed cognition in an aviation landing task (Sorensen et al. 2011), and in aviation and energy to model situations with agents remotely located from each other, (Walker et al. 2010; Salmon et al. 2008 respectively). They are useful to demonstrate the relationship between tasks, technology, and system agents with high face validity. However, the reliability of the method is questionable as different analysts may create different interpretations (Stanton et al. 2013): the method can therefore benefit from application by more than one analyst. OSDs do not require significant training to create and analyse, making them useful for early stage, exploratory analysis of novel systems (Harris et al. 2015).

4.1 OSD development

OSDs were produced for the eight scenarios presented in Table 1. In the diagrams, red arrows are used as a visual indication of the involvement of the RO in the scenario tasks. Red boxes are used to indicate the type of remote operation involved at that moment in the scenario. There is no apparent standard available for OSD symbols, however previous work referenced earlier (Stanton, et al.,. 2022a; Stanton et al. 2022b; Stanton, et al.,. 2021; Banks et al. 2014) has been used as a guide to develop the symbols used here - see Fig. 1.

Fig. 1
figure 1

OSD key (created in serif affinity designer)

Full size image

The diagrams for each scenario show a potential path of interaction between 4 potential scenario agents, illustrating the progression of the scenario until it is resolved in the context of a AV service. The potential scenario agents are: the computer system (CS) which may be the AV system, the system used by the operator, or a background monitoring system operated by the company; the AV User (U); the Remote Operator (RO); and any other External Agents (EA) such as recovery workers or emergency services. The ‘swim lanes’, as referred to in Banks et al. (2014), are categorised by the agent responsible: computing systems (either within the vehicle or RO control centre); AV user; remote operator; and external agents (any other person involved in the scenario other than the AV user and ROs). They are used to clearly represent the different actions of each agent. Arrows show the suggested path through the scenario with red arrows used to indicate when an RO could become involved. Additionally, the expected kind of RO involvement is identified in the OSDs, represented by boxes with the appropriate role indicated.

A single analyst developed the initial OSDs for the eight scenarios. As different analysts with different backgrounds and expertise may produce different solutions for the same task (Stanton et al. 2013) the initial OSDs were iterated based on feedback from two AV experts. The OSD scenarios were presented to two experts in AV Human Factors (both researchers at the University of Nottingham with 5 + years of experience working in AV research) in a collaborative workshop. The two analysts reviewed each of the eight scenarios (Table 1) and independently created their own versions of the OSDs based on their understanding of future AV use. The new OSDs were then compared with the originals and changes made to reflect the additional expert insights. The resulting OSDs are an amalgamation of the expertise of the three analysts: this approach was used to improve the validity of the method, and it is recommended that future application of OSDs follows a similar approach. The two expert analysts were also interviewed on their opinions of the OSD’s that they had produced and on the conclusions they would draw about the roles of the RO from these OSDs. These conclusions were then compared to the initial conclusions drawn from the original OSDs and a general discussion of the findings was had conducted and used in the interpretation of the data, below (Sect. 5.1.2).

Three scenarios, Fig. 2 – Occupant Medical Emergency; Fig. 3 – Weather Disruption Operation and Fig. 4 – Vehicle Collision (Damage Occurs) are presented here to illustrate the OSDs created as part of this research.

4.2 Examples of final version OSDs

These three example scenarios were chosen to give a range of events that may occur with future AV’s, one caused by AV user factors, one caused by uncontrollable external factors and one affecting the fitness of the AV to operate. In these scenarios a ‘perfect’ AV user is included which responds to all requests from the RO as expected. It is important to note that a non-compliant or non-responding vehicle user could result in differences in the tasks and workload of the RO. These three scenarios were also chosen to ensure the presence of all four types of remote operation identified and defined earlier in the paper (RMo, RA, RMa, RD), and to represent expected scenarios from multiple sources of research. These are three of twenty-one scenarios produced as part of this study, although this list is not exhaustive and developments to technology and regulations may affect this.

Fig. 2
figure 2

- Occupant medical emergency (created in serif affinity designer)

Full size image

The scenario of an Occupant Medical Emergency was particularly of interest due to the somewhat isolated nature of travelling in an automated vehicle. As there is no driver to monitor the AV users, a single user who has a medical episode could easily go unnoticed while the vehicle completes its driving programme. This OSD assumes that the emergency has been detected through random checks and confirmed by an RO engaging in RMo. At this point the event enters the request list and is assigned to an RO. The stationary minimal risk condition may not be achieved immediately; it is not intended that the vehicle would immediately stop upon request, as in this instance there is no problem occurring with the vehicle itself. Therefore, the command may be given to remove the vehicle from its current journey and instead to proceed to a safe stopping place at the earliest time available, location either prescribed by the RO or known through AV system local knowledge. Another way of phrasing this may be ‘stopped in a safe place’, reflecting that although there may be places to stop, such as a hard shoulder, these places may not actually be safe for the vehicle occupants or any attending services.

Fig. 3
figure 3

– Weather disrupting operation (created in serif affinity designer)

Full size image

This weather-related scenario was chosen to represent an external situation that is beyond the control of the AV and remote operator. Even with the best design available it is still possible that weather conditions will interfere with vehicle hardware and software and AV operation will be compromised. This also demonstrates a situation where a choice between RMa and RD must occur, although the actual information requirements for this choice are not represented at this stage of research.

Fig. 4
figure 4

Vehicle collision (damage occurs) (created in serif affinity designer)

Full size image

This vehicle collision scenario is intended to cover both collisions by the AV with other vehicles on the road and any other event resulting in damage to the AV as a result of a collision. Due to the damage to the vehicle, it requires a vehicle changeover for the AV user, which may cause communication issues and there is also communication required with multiple external agents, including the later sharing of information in the event of a police or insurance investigation. This scenario also shows the simultaneous work in RMa and RA, and communication between the multiple agents.

4.3 Analysis of OSD’s

In Fig. 2 (occupant medical emergency), the tasks required of an RO completing RMa are quite varied from initial communication and ongoing contact with emergency services to assessing the vehicle stopping point for suitable safety and access. RMa was also the most frequently used RO level in this scenario, with only one task of RMo and one of RA otherwise required. There was no requirement for RD during this scenario, as at no point were the DDTs effected. RMa tasks occurred between the RO and all other scenario agents, emphasizing the importance of developing robust communication channels and providing the necessary infrastructure in the workstation of the RO to facilitate this.

Figure 3 (weather disrupting operation) has a different requirement of communication, as there was no external agent present in the scenario so only communication within the network needs to be considered. The RO is also much more involved in decision making compared to Fig. 2 in which their role is focused on communication with and organisation of emergency services. In the weather scenario the RO is required to make decisions on the information provided to them, which may influence the driving task of the vehicle, either through route planning or RD. For both route planning (RMa) and RD, the RO is required to continually re-evaluate and decide if they need to continue with this task or change until the CS identifies the conditions are suitable for it to resume control of the AV. In the event of weather conditions harsh enough to require RO involvement, the AV user may be uncomfortable, at which point it is appropriate to notify them of the RO involvement and subsequently when the vehicle begins operating fully autonomously again. Again, this diagram is heavily dominated by RMa tasks with RD only occurring after other options have been exhausted. In the event of RD occurring, another ROs may become involved at this point, due to the potential workstation requirements compared to that of an RO undergoing RMa, so the efficient sharing of information is of interest, as well as the differing workstation requirements.

Figure 4 (vehicle collision) again includes all four agent types but only has involvement from the RO on RA and RMa levels. This scenario also includes the involvement of a second vehicle, which may occur whenever there is a terminal issue with the vehicle that would affect its safe operation, or in the case of commercial SAV’s, may occur when the maintenance standards are not suitable for operation. In this scenario, there are several potential outcomes, where the passenger may wish to continue their journey, or not for whatever reason. Re-establishing communication with the AV user, and EMS (if appropriate) in the new vehicle will be important to ensure that it is okay for the vehicle and AV user to leave the scene, as will ensuring communication between the RO and investigative services in the event of accident investigation or insurance claims. The diagram shows the common requirement of RMa and RA occurring simultaneously, or with the task of one supporting the other, and suggesting that, where possible, the RO undertaking RA is also undertaking the RMa tasks. This would require a workstation design to support both these task types and is seen in both the diagrams presented here and in other OSDs produced as part of this work.

Using these three examples, some key insights from the formation of the OSDs have been highlighted and explained. The use of multiple scenarios to investigate the role of the remote operator is important in understanding the great variation in activities which ROs must respond to. Consideration of the three OSDs presented here, as well as the remaining OSDs produced as part of this research, suggests a much greater focus on the tasks involved in RMa and RA in particular is needed, as these appear significantly more often than RD. This is a novel finding which presents remote operation in a new light, in contrast to the earlier views of the RO as taking on DDTs via a remote steering wheel and pedals to simulate traditional driving controls (Kuru 2021), or through other input controls. Other research focuses on investigating different methods of information input, such as using virtual reality headsets (Hosseini and Lienkamp 2016) and haptic feedback to support lateral control (Hosseini et al. 2016). However, due to the well-recognised impact of latency and other communication issues, the workstations themselves of remote operators at all levels have not been widely investigated. Whilst RD still needs to be understood and designed for, this research suggests that RMa and RA will be used in much more frequently to solve the difficult scenarios encountered by AV’s and their users, whilst minimising the influence on the driving task itself. The workstation requirements that ROs will have to support RMa and RA will likely be different to those for RD and this research presents a starting point in identifying these requirements.

4.4 Analysis/consideration of RO type instances

To illustrate the frequency with which the four different RO roles might be undertaken in future scenarios, a brief quantitative analysis was conducted on the eight AV/RO scenarios. This was done by counting the instances of RMo, RA, RMa and RD tasks across the scenarios as shown in Table 2.

Table 2 Number of instances of RMo, RA, RMa and RD across the eight AV/RO scenarios (Percentages give to one decimal place)
Full size table

From this analysis it is possible to see that RD occurs infrequently in comparison to those tasks performed in RA and RMa. However, current research focuses heavily on the provision of RD compared to other methods of remote operation. As RA and RMa make up a significant portion of the potential tasks of an RO there is a strong argument to be made that future research must focus on assistant and management tasks within the RO job role.

There are limitations of the data used here as the OSDs do not show how long would be spent on the different tasks or how significant each of the tasks are in the resolution of issues, which would certainly have an impact on the information requirements for each RO type. Further not all potential scenarios are included. However, as a rough guide to task involvement, this points towards the importance of both RA and RMa in future work design.

5 Discussion and results

The intention of the OSDs, rather than defining the exact path of how these scenarios will be resolved, is to provide examples of the work done to indicate the direction of focus for the design of the future RO job role. Several conclusions could be drawn from the OSDs about the role of an RO. Four types of RO, or RO levels, were identified, forming a hierarchy following an increasing level of RO influence over the AV systems starting with RMo (at the lowest level of influence), then RA, RMa and finally RD (at the highest level).

The starting point for each OSD was the RO being notified of an issue affecting normal operation of the AV. Notifications are triggered by monitoring, which may be performed by computer systems or by the RO (through RMo). RMo may also be used to predict AV operating issues by monitoring for traffic events, weather changes, etc. which may trigger a change in AV operations. This suggests that RMo can be used proactively to reduce the number of reactive RO tasks required. Remote monitoring can also be used on an individual vehicle basis; monitoring of the internal vehicle environment can be used to check for maintenance issues, user safety and possible abuse of the AV. This can be used to address concerns with the safety of passengers, particularly in shared vehicle situations (Wang et al. 2020) and concerns over terrorism or similar events (Liljamo, 2018).

The most commonly appearing roles were that of RA and RMa, likely due to the wide-ranging tasks that may be required from user experience requests to plotting routes around obstacles. They were also noted to often occur simultaneously, with complimentary tasks, i.e., communicating with passengers in the event of an accident whilst simultaneously communicating with emergency services and organising vehicle recovery/replacement. Further research will be required to understand when and how these simultaneous tasks can be managed by a single RO or if multiple personnel are required in some situations to manage workload demands. The emphasis on RA suggests a strong customer-facing role and therefore, ensuring that the RO has access to the training required for customer service tasks, will be a further skill of those RO’s required to engage in RA requests. RA and RMa also appeared as separate tasks in the scenarios, with RMa appearing the most frequently. This reflects the importance of high-level decision-making regarding AV behaviours in these scenarios, without requiring the direct intervention in vehicle control that may be expected by some. In this RMa role in particular the communication with other operators and agents who may be involved in the unfolding scenario (e.g. emergency dispatch, breakdown services, maintenance and recovery) is important in the understanding of the role and design of work requirements.

RD only occurred rarely and after RA and RMa at least had already occurred suggesting that it would be extremely unlikely to be a first solution as the RO would need to move through other stages of operation before being able to decide that RD was appropriate. Indeed, minimising the amount of time that is spent in RD is beneficial for several reasons. Establishing the required situational awareness required to safely drive the vehicle remotely is difficult, as it requires operators to accurately and quickly perceive information about vehicle speed, direction, surrounding conditions (including hazards), the status of AV users and much more. Enabling ROs to successfully acquire all of this information presents a significant workstation design challenge and this role could be subject to a high rate of errors. This also leads into issues of training, to ensure that SA can be achieved, and to ensure that the RO has suitable skills and qualification to remotely drive the vehicle from a remote location. There are also concerns over the communication infrastructure used to support remote driving, for example, if communications to the vehicle fail, or there is a lag in information transmission, this will directly impact remote vehicle control. These same communication channels may also present a risk to the security of the vehicle to cyber-attacks, especially in the event where the vehicle has the capability to be controlled directly from another location. Due to the issues with RD, it is beneficial to consider RD as the highest level of remote operation, where the use of it is strictly limited to only the most critical situations.

5.1 Expert inputs to OSDs

The development stage involving the two AV experts was useful in identifying important areas of the RO role. The task of prioritisation and assignment of assistance requests is of interest, particularly where ROs are managing large numbers of vehicles simultaneously. Effective prioritisation and assignment may be completed by an automated system, although the inclusion of AV users may require a more individual, case-by-case approach requiring RO influence. Prioritisation methods can be seen in situations such as medical care and emergency services dispatch (Napi et al. 2019; Wibring et al. 2022) where computer-based decision support may be used (Gellerstedt et al. 2016). The commercial aspect of AV’s means that request prioritisation may be a balancing act between managing safety and time critical events, such as a road-traffic accidents, with consumer expectations. This also demonstrates an important area for further research in the event of an overload of requests, in a scenario where there is not enough RO’s available to manage the demands from AV’s. Another factor discussed by the experts was balance of control between the RO and the computing systems which may be an area of future work. In some scenarios this is clearer, when the system provides potential solutions and the RO simply selects one, whereas in other scenarios it may be less clear. The experts suggested the range of tasks included in RMa meant it could benefit from being broken down into further sub-roles to ensure the support and training for the RO is sufficient.

RO-AV User communication was a common theme during the discussion. Achieving effective communication among multiple scenario agents simultaneously is of great interest in understanding the role of the RO. Further, the sharing of information between multiple RO’s working on a request, in sequence or in tandem and the requirements of the switching of communication channels, such as in scenarios where the AV users are required to change vehicles will be an area of future research. Much research in AV communication focuses on pedestrian-AV communication, despite there being little consensus on how to achieve it (Rouchitsas and Alm 2019) but there is little research into RO-RO and RO-AV user communication methods. Other further considerations include the potential for user preferences to affect communication, and how the amount of information and way it is presented may affect an AV user. Overall, the same four agents were identified by the experts that were identified by the researcher; external agents were sometimes specified, useful to consider the effect on communication. The external agent may also be another RO involved in the scenario, for example for a different vehicle involved in a multi-vehicle incident, and may not work within the same organisation as the other RO.

6 Limitations and future work

It is important to recognise that for this paper the main focus is not the scenario identification, but the exploration of the roles and tasks of the RO’s within these scenarios. An exhaustive list of scenarios was not produced, rather a set of scenarios which represents a broad range of situation according to the most recent literature and incident reports has been presented here to facilitate investigation of the RO role. Inevitably, a larger set of scenarios would enable further investigation into the roles of the RO. However, it is incredibly difficult to predict scenarios that will emerge from future technology-human interactions, as there will inevitably be interactions which are completely novel. This work does not consider the prioritising of scenarios, and the impact of this prioritisation on RO tasks. Successful prioritisation of tasks may be included under the remit of remote operation, as this task may be completed by an RO, a computer system, or by a combination of both. Further exploration of this may be undertaken to understand the influence of prioritisation on ROs actions. Additionally, this work assumes the presence of a ‘perfect’ AV user, that follows all instructions and responds to requests promptly. Realistically this is unlikely to be the case. The inclusion of an ‘imperfect’ AV user which does not respond to requests and instructions may complicate and change the scenario requirements, but will need to be the focus of future research.

OSD’s are usually created by using data from scenario observation, interviews and HTA’s (Stanton et al. 2013) and validated by comparing to drivers completing simulations of the scenarios and identifying the number of ‘hits’, ‘misses’, ‘false alarms’, and ‘correct rejections’ (Stanton et al. 2021) with a similar method using an on-the-road environment (Stanton et al. 2022b). In the OSDs the AV user is treated as acting ‘perfectly’ in all instances, but this will not always be the case and may impact the resolution process. i.e. by choosing to end the trip unexpectedly, by refusing medical assistance or any other action that is not expected by the OSDs. However, as the scenarios proposed do not yet exist, it is difficult to infer validity or reliability in this way. By involving AV experts to gain insight into additional views on how the OSDs would develop in the chosen scenarios, some validity has been achieved through an abridged type of member-checking. However further studies with simulated scenarios would be appropriate to further confirm the conclusions.

7 Conclusions and recommendations

This paper has presented a set of scenarios for the use of remote operation in autonomous vehicles and proposed a definition for four levels of remote operation. Scenarios were identified from existing research and real-world incident reports to explore these levels of remote operation and further understanding of the requirements for future RO job roles. There are several recommendations that can be made based on this research and these will guide future work in this area:

  1. 1.

    In many AV scenarios tasks will occur quickly and simultaneously and a timely and appropriate response will be required. Research is needed to understand how tasks may need to be prioritised and allocated between human ROs and automated support systems.

  2. 2.

    The prevalence of RA and RMa in comparison to RD, and the conclusion that RD is only used when the other levels have proved insufficient to the task required, suggests that RD is unlikely to be a frequent method of remote operation.

  3. 3.

    Due to the high instances of RA and RMa, compared to the lack of existing research into the workstation requirements for these, suggests that this is an important area for future research to understand the design requirements to allow ROs to complete tasks for these levels of remote operation.

  4. 4.

    The RO is passive until assigned a task; pro-active monitoring may assist in minimising lost time and reduce overall RO requirement, for example, in monitoring weather and traffic conditions to predict possible MRC activations. Research will be required to investigate the effect on RO performance of high levels of monitoring.

  5. 5.

    The methods of communication between the agents in the system are of interest for the RO. Communication with the AV User may occur through several channels depending on the severity of the scenario, and effective communication between the RO, AV User and external agents is required to ensure the transfer of information. Understanding what is needed to facilitate this communication, and ensure efficient transfer of information will also lead into research of the workstation of the RO at different levels of remote operation, as well as providing recommendations for the user interface in the vehicle and communication with external agents such as EMS and police.