Keywords

1 Introduction

Nowadays, highly-automated vehicles that can drive autonomously in specific scenario are conceivable. Drivers in highly-automated vehicle are free from operating the steering wheel, accelerator, or brake, but are requested to supervise the automated driving system and regain control authority when vehicle meets its system limitations such as extremely weather, sensor failure or unpredictable events (International 2016). However, it is highly monotonous for drivers to detect and response to rare and unpredictable events during automated driving, and it requires drivers to keep vigilance for a long time (Hancock 2017). Due to the monotonous process of monitoring, drivers in highly automated vehicle have been shown to become fatigue faster than manual drivers (Schömig et al. 2015; Vogelpohl et al. 2018). Consequently, drivers’ vigilance and the ability of drivers to response to takeover request reduced due to the increase of fatigue level (Greenlee et al. 2018; Körber et al. 2015; Saxby et al. 2013). In addition, low mental workload will impair the performance of driver. Previous studies demonstrated that drivers’ attention resource pools decreased with the reduction in mental workload, and drivers might not aware of the decrease of performance if they were in a state of fatigue (Matthews and Desmond 2002; Young and Stanton 2002). In practice, many methods have been used to detect driver fatigue and drowsiness. In the study of Saito et al. (2016) vibration of vehicle’s lateral position and steering wheel angle were found increasing significantly when the drivers became extremely drowsy. Physiologically, ECG and EEG are widely used to evaluate the fatigue of drivers (Mahachandra et al. 2015; Schmidt et al. 2017). Eye movement metrics were also highly related to the fatigue of drivers. PERCLOS (The percentage of time that the eyelids were closed) was found to be sensitive to driver fatigue (Kozak et al. 2005). In the study of McKinley et al. (2011), an indicator of approximate entropy (ApEn) was found to be more sensitive to evaluate vigilance than PERCLOS. Pupil diameter, blink frequency and closure time were also used to evaluate driver fatigue and vigilance (Abe et al. 2011; Bradley et al. 2008). The studies mentioned above were conducted in the condition of manual driving. However, shortcomings in pilot’s automation monitoring strategies and performance was found based on eye tracking analysis (Sarter et al. 2007). In conditional automated driving condition, drivers are still requested to maintain vigilance for potential dangers and takeover control authority if the automated system meets its system limitations. Due to the totally different driving condition, the eye movement results obtained from manual driving condition may not appropriate for automated driving condition.

Therefore, the eye movement pattern of drivers in conditional automated vehicle is critical for evaluation of driver fatigue and vigilance, and should be explored for better design of in-vehicle HMI. In this study, a driving simulator experiment was conducted in the condition of automated and manual driving.

2 Method

2.1 Participants

Twenty participants (12 men and 8 women) taken part in this experiment had a mean age of 28.5 years (SD = 6.0). The mean driving experience of participants was 5.1 years (SD = 2.7). The participants were students and teachers of the College of automotive engineering, Chongqing University, and all of them had the knowledge of automated driving and driving simulator. All participants received monetary compensation for taking part in the experiment. Participants were instructed to keep their sleep schedule for one week before the experiment, and avoid tea or caffeine on the day of experiment.

2.2 Apparatus

In this experiment, a fixed-base driving simulator (Realtime Technology SimCreator, USA) with automated driving function was adopted. The 180° horizontal field of view was projected by three faceted front screens (1,920 × 1,080 resolution), and the rear views were displayed by three LCD screens. The road noise, engine sound and vehicle vibration were simulated by a series of speakers around the simulator.

The eye movement images of participants were recorded and analyzed through a Dikablis Professional eye tracker (Ergnoeers, Germany). Two infrared cameras were used to record the images of left eye and right eye respectively. The eye movement indicators adopted in this study were calculated by eye movement analysis software D-Lab (Ergnoeers, Germany) based on the images that recorded by the eye tracker.

2.3 Procedure

The study employed a within-subject-design, participants were requested to travel on a monotonous highway with two driving conditions: automated and manual. Before the formal experiment, participants were allowed to familiar themselves with the driving simulator and the automated driving system for 5 min. After that, participants were traveling on a bi-directional four-lines motorway in one of the two conditions. In order to elicit the fatigue of driver, the experiment was conducted in a dark room with dim lights.

Participants were required to monitor the vehicle condition and driving environment, and press a button on the steering wheel immediately when a yellow light on dashboard started to flicker. The time from the light starting to flicker to participants pressing the button was regarded as reaction time. The hint will be displayed on the dashboard for five seconds. In case a participant missed the hint, the reaction time will be regarded as 5 s. The yellow light on dashboard would flicker with an interval of 5 min. After the participants had made response to the yellow light, the subjective fatigue would be evaluated immediately through the Karolinska Sleepiness Scale (KSS) (Åkerstedt and Gillberg 1990). Simultaneously, the data of eye movement in the 5-min period will be averaged. In both automated and manual driving condition, the traffic speeds are restricted to 120 km/h, and the traffic density is 13 vehicles per kilometer. The duration of the experiment was not limited, and it ended until the drivers reached fatigue level of 8. For each participant, the interval of the two experiment is at least one day, and the order was randomized to counterbalance the bias.

2.4 Data Analysis

With the eye movement analysis software D-Lab (Ergnoeers, Germany), evidence values for diverse eye movement could be extracted. In the D-lab, the indicator of percentage of eyelid closure time (PERCLOS) was adopted in this study, which was of interest for objective fatigue evaluation. In addition, the indicator of time of each eye closure (TEEC) were calculated based on the image that recorded by the infrared cameras. The indicators of horizontal eye activity and vertical eye activity were utilized to evaluate the vigilance of drivers. The continuous recording data of eye movement were averaged for each period, and the data of the left and right eye were averaged before further analysis.

A two-factor ANOVA with repeated measures was adopted to analyze the variables separately. One factor is driving condition (two levels) and the repeated measures factor is KSS score: 6, 7 and 8 (K6, K7 and K8). Degrees of freedom were Greenhouse-Geiser corrected, if Mauchly’s test for sphericity showed violated. Post-hoc tests with Bonferroni correction were performed for any statistical significant differences. Statistical significance was noted when p-values were less than 0.05.

3 Results

We measured drivers’ fatigue and vigilance based on subjective and objective metrics. Prior to any inferential analyses, data of eye movement were averaged for the three fatigue levels: K6, K7 and K8.

3.1 Measures of Eye Movement

Horizontal Activity.

The result of horizontal activity was shown in Fig. 1. There was a significant interaction effect between driving condition and fatigue level on horizontal activity \( \left({F\left({2}{,}{38} \right) = 3.306, p = 0.047} \right) \). As for the fatigue level the horizontal activity has no significant difference between the two conditions \( \left( {F\left( {1}{,}{19} \right) = 1.132, p = 0.301} \right) \) for K6, \( F\left( {1{,}19} \right) = 0.057, p = 0.814 \) for K7 and \( F\left({1}{,}{19} \right) = 1.622, p = 0.218 \) for K8). However, in automated driving condition the horizontal activity decreased significantly with the increase of fatigue level \( \left( {F\left( {1.312}{,}{24.937} \right) = 23.381, p\,<\,0.001} \right) \), Post hoc pairwise comparisons indicated that horizontal activity in K7 was significantly lower than K6 (\( p = 0.023 \)) and K8 was significantly lower than K7 \( (p\, < \,0.001) \). In the manual driving condition, the results of ANOVA show that fatigue level had no significant effect on horizontal activity \( \left( {F\left( {2}{,}{38} \right) = 0.146, p = 0.865} \right) \).

Fig. 1.
figure 1

Horizontal eye activities as a function of KSS score for the two driving conditions, error bars (±5) represent standard deviations.

Full size image

Vertical Activity.

The result of vertical activity was shown in Fig. 2. The interaction effect between condition and fatigue level was not significant \( \left( {F\left( {1.407}{,}{26.728} \right) = 2.981, p = 0.083} \right) \). The results of ANOVA analysis demonstrated that drivers’ vertical eye activity had no significant difference between automated and manual driving \( \left( {F\left( {1}{,}{19} \right) = 0.333, p = 0.571} \right) \). However, vertical activity indeed changed with fatigue level \( \left( {F\left( {1.450}{,}{27.555} \right) = 6.257, p = 0.011} \right) \). Post hoc pairwise comparisons indicated that vertical activity in K8 was significantly lower than K6 \( \left( {p = 0.025} \right) \) and K7 \( \left( {p = 0.001} \right) \).

Fig. 2.
figure 2

Vertical eye activities as a function of KSS score for the two driving conditions, error bars (±5) represent standard deviations.

Full size image

PERCLOS.

Figure 3 shows the results of PERCLOS. The results of ANOVA analysis indicate that driving condition and fatigue level have a significant interaction effect on PERCLOS \( (F\left( {1.400}{,}{26.594} \right) = 46.553, p\,<\,0.001) \). In the three fatigue levels, the PERCLOS in automated driving condition was all greater than manual driving condition \( \left( {F\left( {1}{,}{19} \right) = 9.452, p = 0.006} \right) \) for K6, \( F\left( {1}{,}{19} \right) = 31.362, p\,<\,0.001 \) for K7 and \( F\left( {1}{,}{19} \right) = 181.843, p\,<\,0.001 \) for K8). In automated driving condition, the PERCLOS increased significantly with the deepening of the fatigue \( (F\left( {1.240}{,}{23.555} \right) = 57.147, p < 0.001) \). Post hoc pairwise comparisons indicated that PERCLOS in K8 was significantly greater than K7 \( (p\, < \,0.001) \) and K7 was significantly greater than \( {\text{K}}6 (p\, < \,0.001) \). However, in manual driving condition, the PERCLOS seems was not affected by fatigue level \( \left( {F\left( {1.507}{,}{28.639} \right) = 1.056, p = 0.343} \right) \).

Fig. 3.
figure 3

PERCLOS as a function of KSS score for the two driving conditions, error bars represent standard deviations.

Full size image

TEEC.

The results of TEEC was shown in Fig. 4. An ANOVA revealed that period of driving condition and fatigue level had no interaction effect on TEEC \( \left( {F\left( {1.115}{,}{21.951} \right) = 0.289, p = 0.629} \right) \). However, significant difference of TEEC was found between the two conditions \( (F\left( {1}{,}{19} \right) = 19.675, p\,<\,0.001) \). As the Fig. 4 shows, the TEEC increased with the deepening of fatigue in both automated and manual driving condition, but no significant effect was reported \( \left( {F\left( {2}{,}{38} \right) = 3.009, p = 0.061} \right) \).

Fig. 4.
figure 4

TEEC as a function of KSS score for the two driving conditions, error bars represent standard deviations.

Full size image

Reaction Time.

As can be seen in Fig. 5, the reaction time presents out the same change trend with TEEC in both automated and manual driving condition. No interaction effect between driving condition and fatigue level was found \( \left( {F\left( {2}{,}{38} \right) = 0.945, p = 0.397} \right) \). The results of analysis revealed a significant main effect of driving condition, drivers in automated condition indeed made response to hints slower than manual driving \( (F\left( {1}{,}{19} \right) = 18.485, p\,<\,0.001) \). However, the results of ANOVA found that, different from hypothesis, the reaction time didn’t increase with the deepening of fatigue \( \left( {F\left({1.522}{,}{28.910} \right) = 2.322, p = 0.127} \right) \).

Fig. 5.
figure 5

Reaction time as a function of KSS score for the two driving conditions, error bars represent standard deviations.

Full size image

4 Discussion

The purpose of this study is to investigate the difference of eye movement pattern in automated and manual driving condition, and provide reference to in-vehicle HMI design for better safety in conditional automated driving. Previous studies had researched the relationship between eye movement pattern and fatigue level. The study of Jackson et al. (2016) indicated that, the proportion of time with slow eyelid closure was highly related to reaction time, attentional lapses and crashes. The pupil diameter was also found to be related to the fatigue. In some other researches, the pupil diameter decreased with the development of time, and the pupil diameter increased when participants received a short-term cooling which had the effect of relieving fatigue (Schmidt et al. 2017; Schmidt et al. 2017). In addition, the vigilance of driving was also related to the behavior of eyes. In the studies of Abe et al. (2011), increasing in percentage of eyelid closure time and decreasing in blink frequency were observed as the consecutive missed responses increased. However, the studies mentioned above were conducted in manual driving condition. In the research of Abe et al. (2011) the participants were free from driving task, but the event rate was relatively high. The interval between two events was 3 s, and the test will end when closing eyes more than 21 s. However, our experiment was conducted based on the background of conditional automated driving. Actually, the frequency of drivers encountering potentially dangerous events in conditional automated vehicle is quite low. Therefore, the interval between two events in this experiment is 5 min. Due to the monotonous automated driving and underload, drivers got fatigued very soon. In addition, most participants reported that they lost track of time due to the monotonous monitoring task. Therefore, the participants couldn’t be prepared for the next event.

As the results of horizontal eye activity show, when drivers in a state of some signs of sleepiness (K6) the horizontal eye activity is slightly higher in automated driving condition compared with manual driving condition, which is in consistent with the research of Louw and Merat (2017). Drivers in conditional automated driving condition were free from driving tasks, so they had more attention resource for monitoring tasks. In this experiment, participants were driving on a monotonous highway, and the traffic density was 13 vehicles per kilometer. Therefore, the event rate of overtaking was quiet low, which resulted in gazing on the road center. However, despite the drivers seemed to ignore the vehicle dashboard, but they could still make response to the hints on dashboard.

However, a significant decrease in horizontal activity was found with the deepening of fatigue in automated driving condition, but no significant effect was found in manual driving condition. The results of ANOVA demonstrated a significant interaction effect between driving condition and fatigue level. Presumably, drivers can easily see the hint on dashboard and believe they can takeover control authority from automation rapidly.

As for vertical eye activity, no difference was found between the two driving condition. In both automated and manual driving condition, drivers generally check the rearview mirror, and drivers’ attention will be attracted by the vehicles that overtake their vehicle. In spite the fact that the alarm light was placed on the dashboard, drivers were still seldom lowered their gaze to check the alarm light. The main course may be that, the alarm light is in their sight, drivers could still notice the change of alarm light even they didn’t gaze at it.

In this experiment, the PERCLOS and the TEEC in automated driving condition were found significantly higher than manual driving condition in K6, K7 and K8. The indicator of PERCLOS was found to be highly correlated to vigilance while performing visual vigilance tasks (Wierwille and Ellsworth 1994). Additionally, the results of Abe et al. (2011) has shown that the response time increased significantly with the increasing of PERCLOS levels. However, in the automated driving condition experiment of this study, no significant correlation was reported between PERCLOS and reaction time \( \left( {p = 0.636} \right) \). Interestingly, a Pearson correlation analysis revealed that the eye movement indicator of TEEC was highly related to the reaction time \( (p < 0.001) \). In the study of Schmidt et al. (2017), drivers were driving manually, so it is dangerous for drivers to close their eyes for a long period of time. However, this experiment was conducted based on the background of conditional automated driving. Actually, the frequency of drivers encountering potentially dangerous events in conditional automated vehicle is quite low. The reaction time, PERCLOS and the TEEC were increasing simultaneously from the with the deepening of fatigue. However, with the deepening of fatigue, drivers adopted different monitoring strategies for potentially system failure. Participants tended to close their eyes more frequently, instead of prolonging the duration of each eye closure. They tended to take a glance at the driving environment after several seconds of eye closure. This kind of eye movement behavior will increase the PERCLOS rapidly, but the time of each eye closure (TEEC) seemed unaffected. Which is perhaps why, despite the deepening of the fatigue, the driver’s reaction time did not increase significantly with the deepening of fatigue. The study of Lu et al. (2017) concluded that observers could reproduce the layout of a situation in a short period of time. In this experiment, the driving environment and the tasks were both relatively simple. Therefore, taking a glance at driving environment may be sufficient for drivers to keep situation awareness and vigilance for a few seconds.

5 Conclusion

Drivers eye movement pattern in automated and manual driving conditions are different. The eye movement activity of drivers in automated driving vehicle will decrease with the development of fatigue, which results in deteriorated situation awareness and longer reaction time. In addition, the indicator of TEEC is more effective in evaluation of vigilance in conditional automated driving, especially for drivers in extremely sleepy.