Keywords

1 Introduction

The pattern of pilot eye movements can be objectively measured continuously without interfering with the pilot’s activities, through the use of an eye tracker. The visual information captured by eye trackers provides the opportunity to investigate the relationship between eye movement fluctuations and attention shifts while performing tasks (Ahlstrom and Friedman-Berg 2006). Eye scan pattern is one of the methods for assessing a pilot’s cognitive process in the cockpit based on physiological measures (Ayaz et al. 2010). It can provide numerous clues concerning the mental process of encoding information perceived by pilots by using in-flight visual behaviors, such as what areas of interest (AOIs) they scan, dwell and attend (Salvucci and Anderson 1998). Furthermore, eye movements are a sensitive and automatic response which may serve as a window into the process of pilots’ mechanism of situational awareness (SA) and reflect their mental state (Kuo et al. 2009). Endsley (1995) defines three levels of SA, which is linked closely with the major components within cognitive processes. The first level is to perceive environmental cues, such as warning lights in the cockpit. The second level is a process of comprehending the cues based on knowledge and experience. The third level is to predict the possible situation in the near future and project the related measurements to resolve the specific status. SA has been recognized as an essential component within a pilot’s cognitive process in the domain of aviation (Sohn and Doane 2004).

There is an increasing need for further investigation of the relationship between situational awareness performance and the distribution of attention given to the interfaces by the pilot. The bottom-up visual process is stimulus-based and generated from the saliency of information; it can be explained by level one of SA – perception of the cues. On the other side, the top-down approach is a knowledge-based theory that is directed by an internal cognitive process (Henderson 2003). Pilots’ fixation shifts to a specific AOI, in order to acquire the information needed to achieve the task in hand, are controlled by cognitive processes. This complies with the three levels of situational awareness theory of understanding and projection proposed by Endsley (1997). The pilot has to perceive the stimulus in the cockpit, understand the encountered situation, and predict the possible consequences. However, 75 % of pilot errors are resulted from poor perceptual encoding (Jones and Endsley1996). The phenomenon highlights the how interface designs impact pilots’ attention distribution patterns, perceived cognitive load, and SA performance. By utilizing the combination of an eye-tracking device and a flight simulator, pilots’ eye movement patterns, SA performance and perceived workload could be collected for further analysis, and the results could serve as a feedback loop for improving interface design in the future. Therefore, the objectives of the present study are investigating (1) the relationship between interface design and pilots’ SA performance; (2) the relationship between interface design and pilots’ perceived workload; (3) the relationship between pilots’ perceived workload and pilots’ SA performance.

2 Method

2.1 Subjects

There were a total of 69 military pilot participants of in total, 39 Fighter-A type-rated pilots and 30 Fighter-B type-rated pilots. The ages of participants were between 26 and 51 years old, and the total flying hours were between 310 and 3,250 h.

2.2 Apparatus

2.2.1 Flight Simulators

There were two formal fighter simulators, Type A and Type B, used in the experiment. Both of them are high-fidelity training device for military pilots with a fixed-base type consisting of identical cockpit displays to those in the actual aircraft for the purpose of routine flight training and combat planning. The simulators are equipped with a 2-D and 1:1 image projected on a 5-m wide and 3-m high screen. They support pilots’ routine flight training and combat planning. The instructor of both simulators can install scenarios and observe the trainee pilot’s performance via a console with three monitors. The interface designs of Head-up Display (HUD); Integrated Control Panel (ICP); Right Multiple Function Display (RMFD) and Left Multiple Function Display (LMFD) differ slightly in the size and presentation of the symbols and information.

The scenario of this experiment was an air-to-air manoeuvre. The altitude of interceptor (participants) during combat patrol was 20,000 ft with a cruise speed of 300 knots indicated airspeed (KIAS) and heading 050° under the weather conditions of 7-mile visibility and scattered clouds. During a visual rule flight in a patrol route, a foe produced by the simulator unexpectedly would appear at the 9 o’clock position at the same altitude as the interceptor, moving from left to right with heading of 090° and air speed of 300 KIAS. The participants have to search the air space for the foe, and intercept the foe immediately by tactical manoeuvres. In this study, three primary visual behaviours were observed: searching for target with eye contact, pursuing for aiming, and lock-on for pick-off.

2.2.2 Eye Tracking Device

Each pilot’s eye movement patterns were recorded by using a mobile head-mounted eye tracker (ASL Series 4000) designed by Applied Science Laboratory. It is a light (76 g) and portable device allowing subjects to move their head without restriction during the air-to-air maneuvers. Pilots’ eye movements and the related data were collected and stored in a Digital Video Cassette Recorder (DVCR) and then transferred to a computer for further analysis. The definition of a fixation point was three gaze points occurring within an area of 10 by 10 pixels with a 200 ms dwell time (the time spent per glance at an area or display of instrument). The eye movement data in five areas of interest (AOIs) were defined as follows: AOI-1, Head-up Display (HUD); AOI-2, Integrated Control Panel (ICP); AOI-3, Right Multiple Function Display (RMFD); AOI-4, Left Multiple Function Display (LMFD); and AOI-5, Outside of Cockpit (OC). These AOIs provided critical information for pilots performing air-to-air maneuvers.

2.3 Research Design

All subjects (Fighter-A and Fighter-B) undertook the following procedures, (1) complete the demographic data including training experience and total flight hours (5 min); (2) received briefing of the study and the air-to-air scenario (10 min); (3) calibration of the eye tracking device by using three points distributed over the cockpit interface display and screen (10–15 min); (4) perform the air-to-air scenario whilst at the same time an instructor evaluated participants’ situational awareness by activating a warning light during pursuit of the target (3–5 min); (5) evaluate perceived workload by NASA-TLX (5–10 min); and (6) debrief for subject’s feedback (10–15 min).

3 Results

Sixty-nine qualified mission-ready pilots (39 Fighter-A pilots; 30 Fighter-B pilots) participated in this research. Fighter-A pilots had: ages between 26 and 45 years old (M = 34, SD = 5); and total flying hours between 372 and 3,200 h (M = 1294, SD = 753). Fighter-B pilots had: ages between 26 and 51 years old (M = 30, SD = 6); and total flying hours between 310 and 2,920 h (M = 845, SD = 720). The demographic information of all participants’ age, rank, qualification and total flight hours are shown in Table 1. Subjects’ percentage of fixation and average fixation duration across AOIs are shown in Table 2; workload and situational awareness performance are shown in Table 3.

Table 1. Participants’ demographic variables
Full size table
Table 2. T-Test of eye movements at AOIs between Fighter-A and Fighter-B Pilots
Full size table
Table 3. T-Test of participants’ performance of situational awareness and perceived workload between Fighter-A and Fighter-B Pilots.
Full size table

The ‘percentage of fixation’ is proportional data, and it has to be transformed as an arcsine value before further conducting analysis of variance. Table 2 shows that there were significant differences in pilots’ percentage of fixation between the interface designs of Fighter-A and Fighter-B on the HUD (t = 2.66, p < .05, Cohen’s d = .64); ICP (t = 2.09, p < .05, Cohen’s d = .50); RMFD (t = 2.06, p < .05, Cohen’s d = .48); LMFD (t = 2.12, p < .05, Cohen’s d = .47) and OC (t = −3.10, p < .01, Cohen’s d = −.75). Also, there were significant differences in pilots’ fixation duration between the interface designs of Fighter-A and Fighter-B on the HUD (t = 2.93, p < .01, Cohen’s d = .73); ICP (t = 2.05, p < .05, Cohen’s d = .49); LMFD (t = 2.12, p < .05, Cohen’s d = .50) and OC (t = −2.49, p < .05, Cohen’s d = −.60).

The result of pilots’ perceived workload as measured by the NASA-TLX, and the performance of situational awareness pilots achieved are shown in Table 3. Pilots’ workload during the air-to-air flight operation has a significant difference between two different interface designs (t = −4.93, p < .001, Cohen’s d = −1.21). The pilots of Fighter-B show significantly higher perceived workload (M = 61, SD = 15) than Fighter-A pilots (M = 40, SD = 18). In addition, Fighter-B pilots show significantly worse situational awareness performance (M = 0.43, SD = 0.50) compared to Fighter-A pilots (M = 0.85, SD = 0.37).

4 Discussion

With the application of a real-time eye tracking device, interface designers can precisely evaluate the relationship between the cockpit interface and pilots’ performance. In addition, workload might have a negative impact on pilots’ performance thereby increasing the probability of operating hazards. If human factors specialists can detect the human errors which are induced or encouraged by design, early in the design process, they can improve the efficiency of interface design, optimize operator performance, improve aviation safety, and reduce development costs. In the cockpit, pilots’ visual acuity is restricted to only a small area around the fixation point and such the control of eye movements is essential for good situational awareness. Although pilots’ attention can be directed toward a location other than the fixated location in many cases, gaze patterns can be used as a measure of which displayed information pilots are attending to (Johnson and Proctor 2004).

The current research combined eye movement parameters and self-reported workload rating of NASA-TLX to investigate the interface design and pilots’ cognitive demands during an air-to-air task. The findings demonstrated that Fighter-A pilots focus more attention on the HUD than Fighter-B pilots to recognizing the situation confronting them and to plan their strategies to deal with critical situations by seeking information. Furthermore, the salient cues on the HUD might attract pilots’ attention by shifting fixations from outside target to the stimulus, e.g., the malfunction warning alert on the HUD. While perceiving the alert signal with visual scan, pilots need to comprehend the emergency level as possible and project the proper procedures, such as resetting the master caution. Christiansen et al. (1991) found that memory for a scene was related to the number of fixations made on the scene, and that more fixations yielded higher recognition scores. There is a significant difference of Fighter-A pilots’ percentage of fixation on the HUD compared with Fighter-B pilots (45 % vs 36 %), whilst performing the air-to-air task. Furthermore, the average fixation duration of Fighter-A pilots is significantly longer than Fighter-B pilots’ (600 ms vs 457 ms) on the HUD. This demonstrates that Fighter-A pilots spend the highest percentage of fixation on the HUD and pay more attention to conduct cognitive processing of symbols present on the HUD compared with Fighter-B pilots. The interface design of HUD on Fighter-A provides more detailed information to help pilots to perform the air-to-air mission, and pilots have to spend more time on each fixation due to the comprehensive tactical information provided by HUD. As a result Fighter-A pilots could increase their situational awareness by fixating on the HUD. The Fighter-A pilots have significantly higher percentage of fixation (5.2 % vs 2.4 %) and significantly longer fixation duration (189 ms vs 98 ms) on the ICP than Fighter-B pilots. The eye movement patterns on the interfaces of the HUD, ICP, RMFD, LMFD and Outside of Cockpit between Fighter-A pilots and Fighter-B pilots are quite similar; the higher the percentage of fixation, the longer the average fixation duration time. Although there are no significant differences on the percentage of fixation (LMFD) and average fixation duration (RMFD) between the two groups of pilots, it implies that AOIs providing more information would attract pilots’ attention to increase the frequencies of fixations and increase fixation duration in order to decode the visual information by cognitive processes (Table-2).

Compared with Fighter-A pilots, the only AOI for which Fighter-B pilots have a significantly higher percentage of fixation (43 % vs 53 %) and a longer average fixation duration (399 ms vs 460 ms) is Outside of Cockpit. To pursue a dynamic target, pilots have to make an initial visual contact and overlap the target symbol on the HUD for lock-on and pick-off. The fact that Fighter-B pilots allocated a significantly higher percentage of fixation Outside of Cockpit and a significantly smaller percentage of fixation on the HUD might be due to the difference in interface design; the size of the HUD on Fighter-A is bigger than Fighter-B, and the provided information also more dynamic and complex than Fighter-B. This may mean that Fighter-B pilots have to seek additional information for aiming at the target by integrating the information from the HUD with that gained by searching outside of cockpit to make visual contact with the foe. Furthermore, the design difference also increases the cognitive load for Fighter-B pilots, as they have to conduct more information processing to make in-flight decisions to deal with a rapidly changing situation in tactical maneuvers. The phenomenon of different interface designs affecting pilots’ perceived workload can be proven by the results of NASA-TLX, as Fighter-B pilots’ perceived workload is significantly higher than Fighter-A pilots’ perceived workload (Table 3). Strauch (2002) proposed that high workload, competing task demands, and ambiguous cues can all contribute to an operator’s loss of situation awareness, even with experienced and well-trained operators. Furthermore, the findings from the current research also proved that the higher the perceived workload, the poorer the performance in situational awareness. This is a critical issue for aviation safety since Jones and Endsley (1996) found that most information in the cockpit is presented visually and also that over 75 % of pilot errors are caused by perceptual failures due to loss of situational awareness. The findings of the current research could provide the evidence that inappropriate interface design in the cockpit might increase pilots’ workload, and thereby increase the probability of design-induced human errors in flight operations.

5 Conclusion

Understanding a pilot’s visual scan pattern and attention distribution during an air-to-air task will allow aviation professionals to improve the interface designs of advanced cockpits. The present research observed that the interfaces of HUD, ICP, RMFD and LMFD of Fighter-A attract a higher percentage of fixation and longer average fixation durations compared with Fighter-B. Furthermore, Fighter-A pilots’ perceived workloads were lower, but their performance in situational awareness was better than Fighter-B pilots. HUDs in high-tech environments are designed specifically to decrease pilots’ workload by facilitating many cognitive tasks, including information synthesis, monitoring, diagnosis, planning, and prediction, in addition to understanding the physical placement of the fighter. However, the design of information present on the HUD did have a significant impact on pilots’ attention shifts and their situational awareness. Less human-centered interface designs did increase the likelihood that pilots will require more cognitive effort to seek other diagnostic information or process density information in more cognitively complex ways. Eye tracking devices can aid in capturing a pilot’s attention allocation whilst conducting flight simulator trials, and have been proven to be a powerful tool for evaluating the interface design.