Human-Centered Intervention Based on Tactical-Level Input in Unscheduled Takeover Scenarios for Highly-Automated Vehicles

Due to functional limitations in certain situations, the driver receives a request to intervene from automated vehicles operating level 3. Unscheduled intervention of control authority would lead to insufficient situational awareness, then this will make dangerous situations. The purpose of this study is thus to propose tactical-level input (TLI) method with a multimodal driver-vehicle interface (DVI) for the human-centered intervention. The proposed DVI system includes touchscreen, hand-gesture, and haptic interfaces that enable interaction between driver and vehicle, and TLI along with such DVI system can enhance situational awareness. We performed unscheduled takeover experiments using a driving simulator to evaluate the proposed intervention system. The experimental results indicate that TLI can reduce reaction time and driver workload, and moreover, most drivers preferred the use of TLI than manual takeover.


Introduction
The future mobility society would be more harmless, ecological, and convenient by introducing automated driving (AD) technology. Currently, AD can be divided into several levels in terms of degrees of role assignment between the human driver and driving automation system, and six levels defined by SAE International are often used [1]. Level 0 is conventional manual driving (MD) with no automation. Levels 1 and 2 are defined as supportive driving, and require human driver to keep eyes on the road, and perform part of the dynamic driving task. Then, level 3 is defined as conditional automated driving. It does not require the human to keep eyes on the road or hands on the steering wheel, but importantly, human must be ready to takeover control in some situations. The takeover request is usually given 10 s before a functional/domain limitation happens [2]. Finally, levels 4 and 5 are defined as full autonomy that can drive with no human inputs. Here, note that the intermediate levels (level 3), where the human driver essentially required to collaborate with AD system, can be viewed as human-centered autonomy levels.
In systems operating in level 3, the drivers do not need to be vigilant of the surroundings, so they can engage in nondriving related tasks (NDRTs) as long as the AD system operates adequately. When the system predicts that any fault will happen by a functional limitation, it, however, request drivers to intervene, and such intervention is called as takeover. On the basis of the predictability of the situation, scenarios of takeover can be generally classified into scheduled and unscheduled. If the AD system understands the reason of future driver-intervention in advance, such scenario can be regarded as high predictability (scheduled takeover). On the other hand, if the AD system detects functional limitations owing to information from onboard sensors, such scenario has low predictability (unscheduled takeover) [3]. Four example scenarios of unscheduled takeover (defined by the level of current technologies) are shown in Fig. 1. When vehicles operating level 3 run in urban areas, they may request human drivers to intervene within few time period. Unscheduled takeovers require human driver to switch task from NDRT to MD, as shown in Fig. 2. Drivers who engages in NDRTs are not essentially attended to the road environments. The intervention needs an extended period of time for physical and cognitive engagement in the driving task, and suddenly requires attention allocation. This could lead to increasing workload, and may decrease the driving performance, including, lateral and longitudinal control. Therefore, takeovers which is not supported by certain intelligent systems in level 3 (hereinafter manual takeover) potentially have a huge risk.
Typically, unplanned (or emergent) roadworks in urban areas, which include persons using gestures and signs to control traffic, and additive lane signs and markings, result in unscheduled takeover situations. AD systems could detect traffic signs and obstacles, and understand some hand gestures, but they need judgment to choose one from conceivable driving tasks by human [4]. When a vehicle operating level 3 encounters such scenarios, we here hypothesized that using tactical-level input (TLI, explained in detail in subsection 2.2) [5,6] can fulfil safe and effective takeover rather than using operational level control. This is because the AD system can still control the lateral and longitudinal motions even when the AD system falls into a functional limitation. In this study, we performed experiments with 11 participants in a driving simulator, to compare manual takeover and takeover using TLI (hereinafter, TLI takeover). As results of experiments, we could show that TLI enabled safe, smooth, and effective vehicle control as well as decreased driver workload in unscheduled takeovers.
In sum, we evaluate TLI for unscheduled takeover in an urban area. We first integrate a new driver-vehicle interface (DVI) with acoustic and visual feedback functions, to convey the intention of AD system about control transferring, on the basis of the DVI in [7]. We then analyze the driver reaction time, physiological response, and workload in both manual and TLI takeover.  In this section, we clarify the requirements for a DVI for effective unscheduled takeover.

Control Transition and Human Factors
In human-centered automated vehicles, categories of control transition can be divided into four types [8]: The AIDC can be triggered by the limitation of automation system, and the human driver needs to takeover partial or full control in lateral and/or longitudinal directions, within few transition time period. Here, the process of takeover contains the four phases as follows in terms of information processing: & Shift of attention from a NDRT to the driving task & Rapid understanding of the driving situation & Selection of action by utilizing the existing (current) situational awareness & Output of actions for control by the driver In [9][10][11], they found that higher level of automation generally decreased driver's situational awareness and also mental workload. Moreover, they revealed that less time to takeover leads to less chance of mirrors and shoulder checks [12]. In the takeover process, as is well known, the lack of driver situational awareness will lead to dangerous situations.

Categorization of Dynamic Driving Task
Dynamic driving tasks can be generally divided into three levels: operational, tactical, and strategical [5].
& Operational level (MD). The driver controls the steering angle and speed in real-time. & Tactical level. The driver inputs medium-term control commands, including, turning, lane-changing, speed controlling, overtaking, merging, and parking. & Strategical level. The driver inputs long-term trip goals, including, the destination, route, travel time, and driving mode.
If takeover situations occur due to short-term functional limitations, by inputting tactical level command, drivers can fulfill to takeover safely and effectively, rather than simply regressing to MD. The commands of TLI include 'go straight', 'turn right', 'lane-change', 'overtake', and others.
For short-term takeovers, therefore, the risks due to the lack of situational awareness will reduce or eliminate by using TLI, compared to manual takeover.

Required Work: TLI x DVI
A DVI system needs to provide information that enhances the driver's situational awareness, for effectively using TLI in unscheduled takeover situations. The status of the AD system, including, request to intervene, availability of TLI, and information about surroundings, are quite important for the driver to input a command suitable for the situation. A perspective in the human-robot interaction (HRI) field, [13] provides a protocol in human-robot systems to increase situational awareness. Therefore, by referring of the design of the DVI and HRI, we adopted the guidelines as follows.
& The system provides a map to indicate vehicle's path, which are provided by integrated sensor information to reduce the cognitive workload, & The system provides spatial information to make driver recognize vehicle's close surroundings.

Design of Driver-Vehicle Interface
In this section, we explain about the design of the DVI and a feedback system with multimodal channel.

Multimodal DVI for TLI
In this study, we designed a multimodal DVI for TLI, including a touchscreen, hand-gesture, and haptic interfaces, as shown in Fig. 3. The DVI system is connected to the computer of our driving simulator. Drivers can command a TLI by using any of the three types of interfaces. Drivers can select an input modality suited to the driving environment by using a multimodal DVI. In our previous study [7], we confirmed that the DVI system reduced driver workload and input errors, and increased interaction efficiency, compared with unimodal interfaces, and the DVI showed its effectiveness for TLI.

Touchscreen Interface
We developed the touchscreen interface by using Unity platform [14], as shown in Fig. 4 (a). An overview map is displayed on the top left, which shows the ego-vehicle's position and orientation and surroundings. An icon corresponding to ongoing driving mode (top), a sign indicating the next tactical-level maneuver (medium), and the speed limit at the traveling road (bottom) are displayed on top right. By referring to the guidelines in subsection 2.2, a map with integrated sensor information appears in the touchscreen interface. The lateral and longitudinal control buttons are displayed on the bottom of the touchscreen to enable the driver to input tactical level commands.

Hand-Gesture Interface
We then developed a hand-gesture interface based on a motion sensor (Leap Motion). This interface is combined with acoustic and visual feedback to acknowledge or reject driver input.

Haptic Interface
We finally developed the haptic interface. By maneuvering the grip toward lateral and longitudinal directions, drivers can intuitively and rapidly input tactical level commands [6]. Also, the driver can obtain spatial information to enhance situational awareness by force-and tactile-feedback.

DVI for Automated Driving
Two push-buttons, i.e., AD and MD button, attached on the steering wheel was used to switch the driving mode (AD/ MD), as shown in Fig. 4 (c). The AD button engages AD while AD is available. Drivers can engage MD by pressing the MD button. Essentially, the AD system needs to inform the human driver of current status of the AD system, including current driving mode, requests to intervene, availability or unavailability of AD, and system failure. To inform these information in a driving simulator, we integrated visual and acoustic DVIs.

Visual Indication System
We developed the light-emitting diode (LED) DVI which has 120 full-color LEDs controlled by a microcontroller (Arduino), as shown in Fig. 4 (d). We also created icons to indicate different statuses of AD system, as shown in Fig. 4 (b). Here, we explain how the DVI works.
& When a vehicle with AD system is driven by human driver and enters an area where AD is available, the LED illuminates to show its availability, and we adopted a pulsing pattern in "blue for 3 seconds". A corresponding icon simultaneously appears on the touchscreen.  & If the driver turns on AD by pushing AD button, the LED illuminates in brightness-reduced blue and the corresponding icon appears. & When a functional limitation occurs and TLI is available, LED blinks at 2 Hz with orange. When only manual takeover is available, LED blinks at 4 Hz with red.
Different blinking frequency will make even drivers with color-defective vision distinguish the two notifications. Associated with the LEDs, the corresponding icons appear and blink on the touchscreen until driver inputs. During MD mode, the LED turns off, but the corresponding icon appears on the touchscreen.

Acoustic Indication System
During AD mode, drivers may engage in NDRTs, so acoustic interfaces are quite important for the drivers who do not visually attend the road and other DVIs to notify information. The acoustic DVI in the driving simulator has two speakers installed in front of the driver, as shown in Fig. 4 (d). It notifies when functional limitations happen, by playing unique beep sounds at 4 Hz, when TLI and manual takeover is available, respectively.

Design of Experiment
In this section, we describe experimental design including, the driving simulator, functional limitation scenario, participants, and experimental procedure.

Unscheduled Takeover Scenario
We set an unplanned roadwork situation in urban area as the unscheduled takeover scenario and implemented this in the driving simulator [15].

Driving Simulator
We created a 3D virtual environment, which include road modules, road signs, and vehicles. We set the ego-vehicle with a 5-speed automatic transmission system and a 2 lgasoline engine. We can control the dynamic behavior of ego-vehicle (when in AD mode), other vehicles, using scripts written in C# and JavaScript. We also implemented a sensor script to measure distance among vehicles, walls, and other obstacles. We obtained information about the types of obstacles around the vehicle by using Unity's Raycast function.

Scenario
As shown on Fig. 2

Evaluation
Vehicle telemetry, including position, speed, steering angle, pedal position, lane position, and DVI input data was obtained at 100 Hz from the driving simulator. Vehicle telemetry data and DVI control data can bring valuable insight to analyzing driver behavior. Moreover, the driver skin conductance and mean heart rate from E4 wristband [16] were obtained at 4 and 1 Hz, respectively. They can reveal mental stress by analyzing if heart rate accelerates and electrodermal activity (EDA) increases [17]. The accuracy and range are 0.9 nW/ digit (sensor output resolution) for photoplethysmogram (PPG) sensor, and 1 digit per 900 pS (resolution) and 0.01-100 μS (range) for EDA sensor. Furthermore, NASA task load index was obtained to quantify and analyze subjective

Participants and Procedure
In this study, 11 participants took part, and they included 6 males and 5 females (age mean (μ): 28.6 years, standard deviation (σ): 4.2 years). They had 0 to 16 years of experience in driving (μ: 5.3 years, σ: 5.5 years). Six (54.5%) of them had experience of using a driving simulator. All participants had normal or corrected-to-normal vision. Participants were compensated for their contribution. We received informed consent, and then explained about the process of takeover and way to manual takeover as well as TLI takeover to the participants. Then, the participants exercised driving, including takeover procedure, in the driving simulator until they got confidence. To evaluate the proposed system accordingly, we prepared three experiments (one trial per each experiment).
(a) MD from start to end (without AD) (b) AD, then takeover using TLI (TLI takeover) (c) AD, then takeover using MD (manual takeover) For trials (b) and (c), drivers engaged in the NDRT until they receive a request to intervene (RTI). NDRT was here a 2-back cognitive task using a 9-in. tablet (Android), as shown in Fig. 3. After completing the takeover, they switch back to AD mode and to return the NDRT. The order of trials and the location of roadwork was randomized among participants and trials, respectively.

Result and Discussion
In this section, we describe the experimental results, and discuss the implications.

Driver Reaction Time
In this study, the driver reaction time was defined as follows: in manual takeover, the time between RTI and first steering control input of more than 5 degrees, in TLI takeover, the time between RTI and a valid input of a TLI command using the multimodal DVI. The reaction time of each driver and mean reaction time is shown in Figs. 5 (a) and (b), respectively. The average of the driver reaction time in TLI takeover (μ = 4.27, σ = 1.19) was significantly lower than in manual takeover (μ = 6.27, σ = 1.90). The paired two-sample t-test revealed statistical difference between them (p < 0.05, t = 2.96). Thus, we confirm that TLI takeover enables efficient interaction.

Physiological Reaction
Here, we show the driver physiological reactions in terms of skin conductance and heart rate. Due to a signal acquisition error, we had to exclude the data from participants 2, 3, and 10 from this analysis.

Electrodermal Activity
We obtained skin conductance as a measure of driver' EDA.
The maximum values of skin conductance in both TLI and manual takeovers for all drivers are shown in Fig. 6. As stated in subsection 4.2, higher skin conductance can estimate higher cognitive load. The average of the maximum skin conductance values in TLI takeover (μ = 0.521, σ = 0.139) was lower than in manual takeover (μ = 0.708, σ = 0.343), but there was no statistical difference between them. The time variation of skin conductance of participant no. 7 is shown in Fig. 7. We can find from the figure that the skin conductance for TLI takeover was originally lower than in manual takeover. After the AD system issued RTI, the skin conductance for manual takeover increased from 6 s while that for TLI takeover kept calm. These results indicated that TLI made relief and relax even when unscheduled takeover was requested.

Heart Rate
We also recorded the maximum heart rate values for all drivers. The heart rate can be an indication of high workload in both cognitive and physical aspects. Fig. 8 shows that the maximum heart rate for TLI takeover was lower in some drivers than in manual takeover. The variation of heart rate values of participant no. 7 is shown in Fig. 9. The figure shows that the heart rate values for TLI takeover were lower than in manual takeover. However, the average of the maximum heart rate in TLI takeover (μ = 79.8, σ = 7.92) was almost same as that in manual takeover (μ = 79.3, σ = 8.81), and there was no statistical difference between them. We can find that heart rate did not clearly indicate difference but TLI takeover can maintain heart rate values same as that for manual takeover.

Subjective Workload
We obtained NASA task load index from drivers as perceived workload. The average task load scores for each six categories, i.e., mental, physical, temporal, performance, effort, and frustration, are shown in Fig. 10. TLI takeover could decrease subjective workload in all the subscales than those for manual takeover. In particular, scores for physical and effort items were significantly lower, and the paired two-sample t-test revealed statistical difference (physical: p < 0.05, t = 2.36; effort: p < 0.05, t = 3.03) between TLI takeover (physical: μ = 17.7, σ = 19.8; effort: μ = 28.6, σ = 21.5) and manual takeover (physical: μ = 42.7, σ = 29.0; effort: μ = 58.6, σ = 24.8). This result indicates that TLI method can significantly decrease driver workload owing to control intervention.

Questionnaire (Driving Experience)
To investigate driving experience, we asked the participants about preference of using TLI or manual takeover method to intervene in an unscheduled takeover situation. As a result of survey, 90.9% of the drivers mentioned that they prefer to use *: p<0.05 TLI takeover. We could know from the hearing that the major reasons include (a) TLI requires less physical engagement and cognitive attention, (b) efficient and intuitive (easy) input method, and (c) less effort needed for controlling the vehicles. These reasons also corresponds to their subjective workload scores (particularly for physical and effort) described in subsection 5.3 (as shown in Fig. 10).

Advantages of TLI Takeover
We found from experimental results that TLI takeover could significantly reduce the driver reaction time and perceived workloads, compared with manual takeover. We also found that drivers preferred the TLI takeover than manual takeover due to less effort, intuitive, and convenience. This is because the AD system can control the lateral and longitudinal motions even during a takeover situation. On the other hand, some drivers took erroneous behaviors in manual takeover. In unscheduled takeover situations using manual takeover, driver's control performance got significantly worse because drivers had lower or zero situational awareness due to NDRTs.

Discussion: Smoothness
We here need to discuss potential problems of manual takeover (also, in order to emphasize the effectiveness of TLI takeover). We thus analyzed driver performance data, to deeply evaluate the safety issues in manual takeover. We compared data from the trial of MD as the baseline. The time variation of steering angle and speed can reveal the smoothness of both lateral and longitudinal vehicle control. More corrective operations leads to less smoothness, as illustrated in Fig. 11. Here, we can hypothesize that the smoothness relates with driver workload because such operations obviously requires physical and mental effort. As a typical result, Figs. 12 and 13 show the time variation of the steering angle and speed for subject 1 in both MD and manual takeover, respectively. These figures show that both lateral and longitudinal controls are less smooth in manual takeover. The steering control for manual takeover included more corrective operations than that for MD. Moreover, for manual takeover, the driver braked suddenly (decreased the speed) while MD kept a stable speed control. This result indicates that for manual takeover, drivers likely become lack of situational awareness, and then are difficult to input predictive stable operations.
To quantitatively evaluate the performance, we then analyzed the predicted steering angle and speed by performing a second-order Taylor expansion, by referring a method of steering entropy in [18]. To evaluate the control predictability, we derived the prediction error by subtracting the predicted value from the real value. Figs. 14 and 15 show the maximum absolute prediction error of steering angle and speed in both MD and manual takeover. We can find that the average Fig. 14 Maximum absolute prediction error of steering angle prediction error was significantly higher for the speed, and the paired two-sample t-test revealed statistical difference (p < 0.05, t = 2.91) between MD (μ = 1.00, σ = 1.07) and manual takeover (μ = 2.32, σ = 1.04), as shown in Fig. 16 (b). The frequency of the steering entropy is also shown in Fig. 17, and we can find that the steering entropy in manual takeover was higher than that for MD. These results indicate that manual takeover suddenly surges driver workload.

Conclusion and Future Works
Takeover by the human driver in unscheduled situations has high possibility of occurring dangerous situations owing to lack of situational awareness and sudden increase of driver workload. We thus proposed a takeover method using tactical level input (TLI takeover) as a safe and intuitive intervention method for human-centered automated vehicles. We defined an unscheduled takeover scenario in an urban area and performed experiments using a driving simulator. Experimental results indicate that TLI takeover could significantly reduce the driver reaction time and perceived workloads, compared with manual takeover. We also found that drivers preferred the TLI takeover than manual takeover because TLI takeover had advantages in less effort, intuitive, and convenience. The contribution of the paper is to experimentally reveal that TLI method has the potential to make takeover safer and intuitive.
In future work, we will analyze modality and interface suitable for individuals and environments, and integrate a driver monitoring system to estimate driver situational awareness for safe and effective transition of control authority.