1 Introduction

The future mobility society would be more harmless, ecological, and convenient by introducing automated driving (AD) technology. Currently, AD can be divided into several levels in terms of degrees of role assignment between the human driver and driving automation system, and six levels defined by SAE International are often used [1]. Level 0 is conventional manual driving (MD) with no automation. Levels 1 and 2 are defined as supportive driving, and require human driver to keep eyes on the road, and perform part of the dynamic driving task. Then, level 3 is defined as conditional automated driving. It does not require the human to keep eyes on the road or hands on the steering wheel, but importantly, human must be ready to takeover control in some situations. The takeover request is usually given 10 s before a functional/domain limitation happens [2]. Finally, levels 4 and 5 are defined as full autonomy that can drive with no human inputs. Here, note that the intermediate levels (level 3), where the human driver essentially required to collaborate with AD system, can be viewed as human-centered autonomy levels.

In systems operating in level 3, the drivers do not need to be vigilant of the surroundings, so they can engage in non-driving related tasks (NDRTs) as long as the AD system operates adequately. When the system predicts that any fault will happen by a functional limitation, it, however, request drivers to intervene, and such intervention is called as takeover. On the basis of the predictability of the situation, scenarios of takeover can be generally classified into scheduled and unscheduled. If the AD system understands the reason of future driver-intervention in advance, such scenario can be regarded as high predictability (scheduled takeover). On the other hand, if the AD system detects functional limitations owing to information from onboard sensors, such scenario has low predictability (unscheduled takeover) [3]. Four example scenarios of unscheduled takeover (defined by the level of current technologies) are shown in Fig. 1. When vehicles operating level 3 run in urban areas, they may request human drivers to intervene within few time period. Unscheduled takeovers require human driver to switch task from NDRT to MD, as shown in Fig. 2. Drivers who engages in NDRTs are not essentially attended to the road environments. The intervention needs an extended period of time for physical and cognitive engagement in the driving task, and suddenly requires attention allocation. This could lead to increasing workload, and may decrease the driving performance, including, lateral and longitudinal control. Therefore, takeovers which is not supported by certain intelligent systems in level 3 (hereinafter manual takeover) potentially have a huge risk.

Fig. 1
figure 1

Unscheduled takeover scenarios in urban area where are difficult to handle by current automated driving systems

Fig. 2
figure 2

Unscheduled takeover scenario. AD stands for automated driving, NDRT stands for non-driving related task, and TLI stands for task-level input. NDRT is set to a 2-back cognitive-visual task using a table

Typically, unplanned (or emergent) roadworks in urban areas, which include persons using gestures and signs to control traffic, and additive lane signs and markings, result in unscheduled takeover situations. AD systems could detect traffic signs and obstacles, and understand some hand gestures, but they need judgment to choose one from conceivable driving tasks by human [4]. When a vehicle operating level 3 encounters such scenarios, we here hypothesized that using tactical-level input (TLI, explained in detail in subsection 2.2) [5, 6] can fulfil safe and effective takeover rather than using operational level control. This is because the AD system can still control the lateral and longitudinal motions even when the AD system falls into a functional limitation. In this study, we performed experiments with 11 participants in a driving simulator, to compare manual takeover and takeover using TLI (hereinafter, TLI takeover). As results of experiments, we could show that TLI enabled safe, smooth, and effective vehicle control as well as decreased driver workload in unscheduled takeovers.

In sum, we evaluate TLI for unscheduled takeover in an urban area. We first integrate a new driver-vehicle interface (DVI) with acoustic and visual feedback functions, to convey the intention of AD system about control transferring, on the basis of the DVI in [7]. We then analyze the driver reaction time, physiological response, and workload in both manual and TLI takeover.

2 Related and Required Work

In this section, we clarify the requirements for a DVI for effective unscheduled takeover.

2.1 Control Transition and Human Factors

In human-centered automated vehicles, categories of control transition can be divided into four types [8]:

  • DIDC: Driver-initiated driver control

  • DIAC: Driver-initiated automation control

  • AIDC: Automation-initiated driver control

  • AIAC: Automation-initiated automation control

The AIDC can be triggered by the limitation of automation system, and the human driver needs to takeover partial or full control in lateral and/or longitudinal directions, within few transition time period. Here, the process of takeover contains the four phases as follows in terms of information processing:

  • Shift of attention from a NDRT to the driving task

  • Rapid understanding of the driving situation

  • Selection of action by utilizing the existing (current) situational awareness

  • Output of actions for control by the driver

In [9,10,11], they found that higher level of automation generally decreased driver’s situational awareness and also mental workload. Moreover, they revealed that less time to takeover leads to less chance of mirrors and shoulder checks [12]. In the takeover process, as is well known, the lack of driver situational awareness will lead to dangerous situations.

2.2 Categorization of Dynamic Driving Task

Dynamic driving tasks can be generally divided into three levels: operational, tactical, and strategical [5].

  • Operational level (MD). The driver controls the steering angle and speed in real-time.

  • Tactical level. The driver inputs medium-term control commands, including, turning, lane-changing, speed controlling, overtaking, merging, and parking.

  • Strategical level. The driver inputs long-term trip goals, including, the destination, route, travel time, and driving mode.

If takeover situations occur due to short-term functional limitations, by inputting tactical level command, drivers can fulfill to takeover safely and effectively, rather than simply regressing to MD. The commands of TLI include ‘go straight’, ‘turn right’, ‘lane-change’, ‘overtake’, and others. For short-term takeovers, therefore, the risks due to the lack of situational awareness will reduce or eliminate by using TLI, compared to manual takeover.

2.3 Required Work: TLI x DVI

A DVI system needs to provide information that enhances the driver’s situational awareness, for effectively using TLI in unscheduled takeover situations. The status of the AD system, including, request to intervene, availability of TLI, and information about surroundings, are quite important for the driver to input a command suitable for the situation. A perspective in the human-robot interaction (HRI) field, [13] provides a protocol in human-robot systems to increase situational awareness. Therefore, by referring of the design of the DVI and HRI, we adopted the guidelines as follows.

  • The system provides a map to indicate vehicle’s path, which are provided by integrated sensor information to reduce the cognitive workload,

  • The system provides spatial information to make driver recognize vehicle’s close surroundings.

3 Design of Driver-Vehicle Interface

In this section, we explain about the design of the DVI and a feedback system with multimodal channel.

3.1 Multimodal DVI for TLI

In this study, we designed a multimodal DVI for TLI, including a touchscreen, hand-gesture, and haptic interfaces, as shown in Fig. 3. The DVI system is connected to the computer of our driving simulator. Drivers can command a TLI by using any of the three types of interfaces. Drivers can select an input modality suited to the driving environment by using a multimodal DVI. In our previous study [7], we confirmed that the DVI system reduced driver workload and input errors, and increased interaction efficiency, compared with unimodal interfaces, and the DVI showed its effectiveness for TLI.

Fig. 3
figure 3

Driving simulator and three types of driver-vehicle interface including touchscreen, gesture, and haptic

3.1.1 Touchscreen Interface

We developed the touchscreen interface by using Unity platform [14], as shown in Fig. 4 (a). An overview map is displayed on the top left, which shows the ego-vehicle’s position and orientation and surroundings. An icon corresponding to ongoing driving mode (top), a sign indicating the next tactical-level maneuver (medium), and the speed limit at the traveling road (bottom) are displayed on top right. By referring to the guidelines in subsection 2.2, a map with integrated sensor information appears in the touchscreen interface. The lateral and longitudinal control buttons are displayed on the bottom of the touchscreen to enable the driver to input tactical level commands.

Fig. 4
figure 4

Driver-vehicle interaction system with visual and acoustic feedback for automated driving. (a) Touchscreen interface, (b) icons, (c) steering wheel DVI with AD and MD buttons, (d) ambient light DVI, (e) speaker, and (f) NDRT tablet

3.1.2 Hand-Gesture Interface

We then developed a hand-gesture interface based on a motion sensor (Leap Motion). This interface is combined with acoustic and visual feedback to acknowledge or reject driver input.

3.1.3 Haptic Interface

We finally developed the haptic interface. By maneuvering the grip toward lateral and longitudinal directions, drivers can intuitively and rapidly input tactical level commands [6]. Also, the driver can obtain spatial information to enhance situational awareness by force- and tactile-feedback.

3.2 DVI for Automated Driving

Two push-buttons, i.e., AD and MD button, attached on the steering wheel was used to switch the driving mode (AD/MD), as shown in Fig. 4 (c). The AD button engages AD while AD is available. Drivers can engage MD by pressing the MD button. Essentially, the AD system needs to inform the human driver of current status of the AD system, including current driving mode, requests to intervene, availability or unavailability of AD, and system failure. To inform these information in a driving simulator, we integrated visual and acoustic DVIs.

3.2.1 Visual Indication System

We developed the light-emitting diode (LED) DVI which has 120 full-color LEDs controlled by a microcontroller (Arduino), as shown in Fig. 4 (d). We also created icons to indicate different statuses of AD system, as shown in Fig. 4 (b). Here, we explain how the DVI works.

  • When a vehicle with AD system is driven by human driver and enters an area where AD is available, the LED illuminates to show its availability, and we adopted a pulsing pattern in “blue for 3 seconds”. A corresponding icon simultaneously appears on the touchscreen.

  • If the driver turns on AD by pushing AD button, the LED illuminates in brightness-reduced blue and the corresponding icon appears.

  • When a functional limitation occurs and TLI is available, LED blinks at 2 Hz with orange. When only manual takeover is available, LED blinks at 4 Hz with red.

Different blinking frequency will make even drivers with color-defective vision distinguish the two notifications. Associated with the LEDs, the corresponding icons appear and blink on the touchscreen until driver inputs. During MD mode, the LED turns off, but the corresponding icon appears on the touchscreen.

3.2.2 Acoustic Indication System

During AD mode, drivers may engage in NDRTs, so acoustic interfaces are quite important for the drivers who do not visually attend the road and other DVIs to notify information. The acoustic DVI in the driving simulator has two speakers installed in front of the driver, as shown in Fig. 4 (d). It notifies when functional limitations happen, by playing unique beep sounds at 4 Hz, when TLI and manual takeover is available, respectively.

4 Design of Experiment

In this section, we describe experimental design including, the driving simulator, functional limitation scenario, participants, and experimental procedure.

4.1 Unscheduled Takeover Scenario

We set an unplanned roadwork situation in urban area as the unscheduled takeover scenario and implemented this in the driving simulator [15].

4.1.1 Driving Simulator

We created a 3D virtual environment, which include road modules, road signs, and vehicles. We set the ego-vehicle with a 5-speed automatic transmission system and a 2 l- gasoline engine. We can control the dynamic behavior of ego-vehicle (when in AD mode), other vehicles, using scripts written in C# and JavaScript. We also implemented a sensor script to measure distance among vehicles, walls, and other obstacles. We obtained information about the types of obstacles around the vehicle by using Unity’s Raycast function.

4.1.2 Scenario

As shown on Fig. 2, a roadwork closes a part of the lane, so vehicles that travels on that lane (left lane) require to move to the next lane (right lane). The vehicle completes to pass the roadwork zone, and then moves back to the original lane.

4.2 Evaluation

Vehicle telemetry, including position, speed, steering angle, pedal position, lane position, and DVI input data was obtained at 100 Hz from the driving simulator. Vehicle telemetry data and DVI control data can bring valuable insight to analyzing driver behavior. Moreover, the driver skin conductance and mean heart rate from E4 wristband [16] were obtained at 4 and 1 Hz, respectively. They can reveal mental stress by analyzing if heart rate accelerates and electrodermal activity (EDA) increases [17]. The accuracy and range are 0.9 nW/digit (sensor output resolution) for photoplethysmogram (PPG) sensor, and 1 digit per 900 pS (resolution) and 0.01–100 μS (range) for EDA sensor. Furthermore, NASA task load index was obtained to quantify and analyze subjective workloads. Another questionnaire was obtained to analyze driving experience of participants after they completed all the trials.

4.3 Participants and Procedure

In this study, 11 participants took part, and they included 6 males and 5 females (age mean (μ): 28.6 years, standard deviation (σ): 4.2 years). They had 0 to 16 years of experience in driving (μ: 5.3 years, σ: 5.5 years). Six (54.5%) of them had experience of using a driving simulator. All participants had normal or corrected-to-normal vision. Participants were compensated for their contribution.

We received informed consent, and then explained about the process of takeover and way to manual takeover as well as TLI takeover to the participants. Then, the participants exercised driving, including takeover procedure, in the driving simulator until they got confidence. To evaluate the proposed system accordingly, we prepared three experiments (one trial per each experiment).

  1. (a)

    MD from start to end (without AD)

  2. (b)

    AD, then takeover using TLI (TLI takeover)

  3. (c)

    AD, then takeover using MD (manual takeover)

For trials (b) and (c), drivers engaged in the NDRT until they receive a request to intervene (RTI). NDRT was here a 2-back cognitive task using a 9-in. tablet (Android), as shown in Fig. 3. After completing the takeover, they switch back to AD mode and to return the NDRT. The order of trials and the location of roadwork was randomized among participants and trials, respectively.

5 Result and Discussion

In this section, we describe the experimental results, and discuss the implications.

5.1 Driver Reaction Time

In this study, the driver reaction time was defined as follows: in manual takeover, the time between RTI and first steering control input of more than 5 degrees, in TLI takeover, the time between RTI and a valid input of a TLI command using the multimodal DVI. The reaction time of each driver and mean reaction time is shown in Figs. 5 (a) and (b), respectively. The average of the driver reaction time in TLI takeover (μ = 4.27, σ = 1.19) was significantly lower than in manual takeover (μ = 6.27, σ = 1.90). The paired two-sample t-test revealed statistical difference between them (p < 0.05, t = 2.96). Thus, we confirm that TLI takeover enables efficient interaction.

Fig. 5
figure 5

Driver reaction time for each participant and average in using TLI and manual takeover

5.2 Physiological Reaction

Here, we show the driver physiological reactions in terms of skin conductance and heart rate. Due to a signal acquisition error, we had to exclude the data from participants 2, 3, and 10 from this analysis.

5.2.1 Electrodermal Activity

We obtained skin conductance as a measure of driver’ EDA. The maximum values of skin conductance in both TLI and manual takeovers for all drivers are shown in Fig. 6. As stated in subsection 4.2, higher skin conductance can estimate higher cognitive load. The average of the maximum skin conductance values in TLI takeover (μ = 0.521, σ = 0.139) was lower than in manual takeover (μ = 0.708, σ = 0.343), but there was no statistical difference between them. The time variation of skin conductance of participant no. 7 is shown in Fig. 7. We can find from the figure that the skin conductance for TLI takeover was originally lower than in manual takeover. After the AD system issued RTI, the skin conductance for manual takeover increased from 6 s while that for TLI takeover kept calm. These results indicated that TLI made relief and relax even when unscheduled takeover was requested.

Fig. 6
figure 6

Maximum skin conductance for each participant and average

Fig. 7
figure 7

Time variation of skin conductance (subject 7)

5.2.2 Heart Rate

We also recorded the maximum heart rate values for all drivers. The heart rate can be an indication of high workload in both cognitive and physical aspects. Fig. 8 shows that the maximum heart rate for TLI takeover was lower in some drivers than in manual takeover. The variation of heart rate values of participant no. 7 is shown in Fig. 9. The figure shows that the heart rate values for TLI takeover were lower than in manual takeover. However, the average of the maximum heart rate in TLI takeover (μ = 79.8, σ = 7.92) was almost same as that in manual takeover (μ = 79.3, σ = 8.81), and there was no statistical difference between them. We can find that heart rate did not clearly indicate difference but TLI takeover can maintain heart rate values same as that for manual takeover.

Fig. 8
figure 8

Maximum heart rate for each participant and average

Fig. 9
figure 9

Time variation of heart rate (subject 7)

5.3 Subjective Workload

We obtained NASA task load index from drivers as perceived workload. The average task load scores for each six categories, i.e., mental, physical, temporal, performance, effort, and frustration, are shown in Fig. 10. TLI takeover could decrease subjective workload in all the subscales than those for manual takeover. In particular, scores for physical and effort items were significantly lower, and the paired two-sample t-test revealed statistical difference (physical: p < 0.05, t = 2.36; effort: p < 0.05, t = 3.03) between TLI takeover (physical: μ = 17.7, σ = 19.8; effort: μ = 28.6, σ = 21.5) and manual takeover (physical: μ = 42.7, σ = 29.0; effort: μ = 58.6, σ = 24.8). This result indicates that TLI method can significantly decrease driver workload owing to control intervention.

Fig. 10
figure 10

Subjective task load score for each subscale and average

5.4 Questionnaire (Driving Experience)

To investigate driving experience, we asked the participants about preference of using TLI or manual takeover method to intervene in an unscheduled takeover situation. As a result of survey, 90.9% of the drivers mentioned that they prefer to use TLI takeover. We could know from the hearing that the major reasons include (a) TLI requires less physical engagement and cognitive attention, (b) efficient and intuitive (easy) input method, and (c) less effort needed for controlling the vehicles. These reasons also corresponds to their subjective workload scores (particularly for physical and effort) described in subsection 5.3 (as shown in Fig. 10).

5.5 Advantages of TLI Takeover

We found from experimental results that TLI takeover could significantly reduce the driver reaction time and perceived workloads, compared with manual takeover. We also found that drivers preferred the TLI takeover than manual takeover due to less effort, intuitive, and convenience. This is because the AD system can control the lateral and longitudinal motions even during a takeover situation. On the other hand, some drivers took erroneous behaviors in manual takeover. In unscheduled takeover situations using manual takeover, driver’s control performance got significantly worse because drivers had lower or zero situational awareness due to NDRTs.

6 Discussion: Smoothness

We here need to discuss potential problems of manual takeover (also, in order to emphasize the effectiveness of TLI takeover). We thus analyzed driver performance data, to deeply evaluate the safety issues in manual takeover. We compared data from the trial of MD as the baseline. The time variation of steering angle and speed can reveal the smoothness of both lateral and longitudinal vehicle control. More corrective operations leads to less smoothness, as illustrated in Fig. 11. Here, we can hypothesize that the smoothness relates with driver workload because such operations obviously requires physical and mental effort.

Fig. 11
figure 11

Driver’s control performance measures including steering angle and pedal position

As a typical result, Figs. 12 and 13 show the time variation of the steering angle and speed for subject 1 in both MD and manual takeover, respectively. These figures show that both lateral and longitudinal controls are less smooth in manual takeover. The steering control for manual takeover included more corrective operations than that for MD. Moreover, for manual takeover, the driver braked suddenly (decreased the speed) while MD kept a stable speed control. This result indicates that for manual takeover, drivers likely become lack of situational awareness, and then are difficult to input predictive stable operations.

Fig. 12
figure 12

Time variation of steering angle (subject 1)

Fig. 13
figure 13

Time variation of speed (subject 1)

To quantitatively evaluate the performance, we then analyzed the predicted steering angle and speed by performing a second-order Taylor expansion, by referring a method of steering entropy in [18]. To evaluate the control predictability, we derived the prediction error by subtracting the predicted value from the real value. Figs. 14 and 15 show the maximum absolute prediction error of steering angle and speed in both MD and manual takeover. We can find that the average prediction error was significantly higher for the speed, and the paired two-sample t-test revealed statistical difference (p < 0.05, t = 2.91) between MD (μ = 1.00, σ = 1.07) and manual takeover (μ = 2.32, σ = 1.04), as shown in Fig. 16 (b). The frequency of the steering entropy is also shown in Fig. 17, and we can find that the steering entropy in manual takeover was higher than that for MD. These results indicate that manual takeover suddenly surges driver workload.

Fig. 14
figure 14

Maximum absolute prediction error of steering angle

Fig. 15
figure 15

Max absolute prediction error of speed

Fig. 16
figure 16

Average prediction error of steering wheel and speed

Fig. 17
figure 17

Steering entropy for manual driving and manual takeover

7 Conclusion and Future Works

Takeover by the human driver in unscheduled situations has high possibility of occurring dangerous situations owing to lack of situational awareness and sudden increase of driver workload. We thus proposed a takeover method using tactical level input (TLI takeover) as a safe and intuitive intervention method for human-centered automated vehicles. We defined an unscheduled takeover scenario in an urban area and performed experiments using a driving simulator. Experimental results indicate that TLI takeover could significantly reduce the driver reaction time and perceived workloads, compared with manual takeover. We also found that drivers preferred the TLI takeover than manual takeover because TLI takeover had advantages in less effort, intuitive, and convenience. The contribution of the paper is to experimentally reveal that TLI method has the potential to make takeover safer and intuitive.

In future work, we will analyze modality and interface suitable for individuals and environments, and integrate a driver monitoring system to estimate driver situational awareness for safe and effective transition of control authority.