1 Introduction

Recent advancements in augmented reality (AR) technology present innovative and immersive approaches to expressing human computer interactions, providing sophisticated applications to support a variety of complex human activities. AR provides a layer of digital information that is overlaid onto real world elements, enabling the real world to be augmented by computer-generated 2D and 3D digital objects. AR technologies are gaining importance in multiple domains including healthcare, manufacturing, and education and are changing the way we work in these fields (Park et al. 2021). Chien et al. (2019) developed AR symbology to assist surgeons, superimposing medical data onto the physical surface of patients. Likewise, in the engineering domain, Szajna et al. (2020) demonstrated that AR symbology assisted the wiring of control cabinets in the assembly and production process.

Interest in AR technology has been expressed by the aviation industry, with an emphasis on utilising AR to support engineering, strategic navigation, and training and simulation (Safi et al. 2019). AR cockpit solutions are being explored within programmes that are developing eye-visor technology for civil fixed wing (Alvarez and Rodriguez 2021) and rotary wing cockpits (Walko 2018), to meet the requirements of complex future flight concepts (Blundell et al. 2019; Blundell, Huddlestone, et al. 2020). Regardless, the design and development requirements to enact such design changes, whilst maintaining the highest levels of safety, are substantial. Unfortunately, whilst existing head-mounted display (HMD) guidelines relevant to AR (e.g. Federal Aviation Administration AC 25-11B, 2014) have a firm foundation in human factors and perception (Melzer et al. 2009), the rapid advance in technology means the design guidance for a lightweight HMDs for commercial aviation, urban air mobility platforms, and AR applications in general, has not kept pace (Ariansyah et al. 2022). Even the most recent and sophisticated aviation HMDs, such as the BAE Systems Striker II HMD used in the Eurofighter Typhoon II or the Collins Gen III Helmet Mounted Display System (HMDS) used in the Lockheed-Martin F35, employ symbology that has largely been adapted from head-up-displays (HUD), an HMD precursor, rather than implementing eye/head-referenced symbology that can be fully exploited with HMD AR technology.

Across industries, the pertinent challenge of how to best present AR imagery has largely been addressed through development of hardware capabilities, tracking performance, and the accessibility of content creation software (Bottani and Vignali 2019; Palmarini et al. 2018). Human factors (HFs) considerations are critical for answering what AR applications should display to ensure efficiency and effectiveness (Akçayir et al. 2016; Poushneh 2018). Despite the recognised importance of HF studies in the AR literature, user-based experiments examining different interface designs, information requirements, display types, etc. and their impacts on task performance, ergonomics, and safety are limited (Ariansyah et al. 2022). In particular, information requirements are key for AR researchers and practitioners to understand the cognitive and ergonomic needs of the end user. Hence, a user-centred design (UCD) approach exploring how AR technology can enhance pilot decision making in commercial aviation environments will enhance the usability, trust, and acceptance of AR technology more widely.

This section will focus first on providing a brief history of AR in commercial flight decks, followed by an overview of the UCD approach and the theoretical framework in which the current study was conducted.

1.1 Literature review

1.1.1 History of AR in commercial flight deck design

Early HUDs, in the form of a gyroscopic gunsight during World War II (Fig. 1a), that superimposed digital targeting information upon the user’s forward field of view, can be considered the progenitor of modern AR. Gyroscopic sights were supplemented with other navigational and spatial related AR symbology in the 1950s, with the first HUD-equipped Royal Navy Buccaneer entering service in 1962 (Nichol 2015). By the 1970s, HUDs became a mainstay of the military fighter jet, enabling pilots to simultaneously perceive flight information (e.g. attitude, air data and navigational guidance) and the external forward view of the natural scene (Fig. 1). Over subsequent decades, HUD symbology complexity slowly increased as a function of growth in computing power and the real-time processing of accurate global positioning information. The latest AR advancements for HUDs have seen the integration of data from a wide range of sensors allowing sophisticated synthetic terrain or infrared video information (Fig. 1d).

Fig. 1
figure 1

(Source: Wikipedia (CC)/FAA)

Technological advancement of the HUD: a Reflector gunsight used during World War II; b Modern military HUD on F-16; c Civil Airliner HUD on the Boeing 787; d Enhanced flight vision system on Dassault Falcon 900LX.

Full size image

HMD (or head-worn display (HWD)) technology, which has its origins in the military domain, emerged in the 1960s as monocular displays. In the 1970s, an electro-mechanical linkage head tracked system, that provided the pilot with a superimposed helmet mounted reticule, was deployed in the U.S. Army’s AH-1G Huey Cobra attack helicopter (Bayer et al. 2009). Likewise, the National Aeronautics and Space Administration (NASA) developed the monocular night vision system for the Boeing AH-64 Apache helicopter in the 1980s (Rash and Martin 1988). The use of binocular HMDs has become standard on modern military fast jets (e.g. the Lockheed-Martin F-35 and the Eurofighter Typhoon II) and will be the primary enabler of future AR and VR military display concepts being developed across multiple nations (e.g. the BAE Systems Tempest cockpit (Fig. 2)). Compared to HUDs, where the fixed forward location limits the field of regard, HMD symbology can be head or eye referenced, and world or aircraft referenced (permitting an unlimited field of regard) when coupled with low latency head-trackers (Yeh et al. 2003). Research on AR HMDs for aeronautical engineering, planning air traffic flows, and training and simulation has also been undertaken (Safi et al. 2019). The pursuit of a lightweight AR HMD (i.e. eye-visor technology) is being carried out across multiple research and development programmes (Alvarez and Rodriguez,2021), focusing on the presentation of task specific information to enhance pilot/automation interaction and reduce workload.

Fig. 2
figure 2

(Source: BAE Systems)

Potential AR and VR application of future jet fighters—the BAE Systems Tempest: a Striker II HMD used on the Eurofighter Typhoon II; b—Illustrated concept of the Tempest AR/VR cockpit; c—Tempest cockpit prototype as presented via the Striker II HMD.

Full size image

Some HUD and HMD symbology can be considered forms of AR imagery. Notably, AR’s significant benefit has been the enhancement of spatial awareness and safety during low-visibility, low-altitude operations through the capability to make digital or degraded visual information visible, for instance, during low-visibility helicopter approaches (Stanton et al. 2019) or ground taxiing operations (Arthur III et al. 2014; Blundell et al. 2023). A prime AR example is the “highway in the sky” (HITS, Fig. 3), a tunnel display consisting of discrete floating gates depicting a perspective view of the aircraft’s pre-planned flight trajectory. Recent research (Li et al. 2020; Tran et al. 2018) found that the application and benefits of HMD presented AR symbology is not exclusive to the depiction of external information. Specifically, critical decision making can be enhanced with symbology that highlights and provides guidance on relevant cockpit displays. However, many studies investigating HMD applications on the flight deck have underlined the usability burdens associated with HMD weight and the encumbrance that impinge on operator comfort (Arthur III et al. 2014; Tran et al. 2018), potentially negating the HMD’s wider field of regard advantages.

Fig. 3
figure 3

(Source: NASA)

Classical aviation AR applications in the form of path following guidance provided from a highway in the sky (HITS). Presented here within a flight simulator.

Full size image

The cognitive benefits (and detriments) of AR symbology presented via a HUD in the aviation domain have long been documented and are relatively mature compared to recent AR human factors studies. For example, HUD symbology has afforded enhanced situation awareness (Fadden et al. 2001; Wickens and Alexander 2009), particularly when the digital information is geographically mapped to locations (i.e. the symbology is conformal within the user’s perspective visual scene). Conversely, HUD presented AR can lead to the user allocating attention to the AR imagery at the expense of other critical task-related visual information (Fadden et al. 1998; Stuart et al. 2001). In contrast, only in the past decade have similar attentional capture effects of AR been reported in other domains, notably within medicine (Dixon et al. 2012, 2014). Solutions offered here have included using “wire-mesh” digital objects in place of solid overlays to address attentional issues imposed by AR (Guha et al. 2017; Marcus et al. 2015). Lessons learned from aviation HUD research, however, suggests that the visual occlusion caused by solid overlays is not the root cause of AR attentional issues. For instance, the earliest evidence of AR attentional capture in aviation was recorded using a wire-mesh highway-in-the-sky display (Fischer et al. 1980).

1.1.2 User-centred design requirement elicitation

Users are often overlooked within the initial stages of technology-centred design processes, resulting in final designs with less effective system safety and performance (Stanton et al. 2017). UCD emphasises the early inclusion of users to generate useful requirements for initial design concepts, which improves user acceptance and reduces the likelihood of design-induced errors (Baber and Mellor 2001). Examples of approaches to user-requirement elicitation include conducting interviews and focus groups with subject matter experts (SME) to elicit opinions and preferences concerning the target technology. User inputs are then analysed using thematic analysis (TA) (Braun and Clarke 2006) to capture the requirements for the initial design concept. TA is suited to describing design requirements as it permits user discourse to be coded using a combination of deductive ‘top-down’ or inductive ‘bottom-up’ approaches. In recent design requirements studies, TA has been employed effectively in medical, human–computer interface, environmental construction, and business domains (Agyekum et al. 2019; Babar et al. 2018; Bouamrane et al. 2019).

1.1.3 Designing for situation awareness

Endsley’s three-level model of situation awareness (SA) (Endsley 1988), provides a framework to capture the role that pilot and system interactions have on pilot decision making. Endsley’s model presents a straightforward approach to pinpointing current weaknesses in existing system design, and for identifying where the optimisation of task goals might be aligned with the capabilities of AR technology. The utility of the model in supporting the design of easy-to-interpret displays has been established across many application areas, including aviation, medicine, and cyber security (Endsley 2001; Schulz et al. 2013; Wickens et al. 2014). The model consists of three levels of situation awareness that must be applied to a specific situation: Perception (level 1)—perceiving the status, attributes, and dynamics of relevant elements in a system. For a pilot, this would include critical elements such as the terrain, proximal aircraft, warning lights; Comprehension (level 2)—involves the interpretation and understanding of the meaning of disjointed level 1 elements, and Projection (level 3)—predicting the implications of level 1 and 2 information for the future. Applying the model within a UCD approach permits the systematic evaluation of how AR might enhance user awareness of information across different SA levels.

1.2 Study objective

The current study addresses the existing literature gap concerning the sparsity of aviation specific and general AR design guidance. Specifically, the study focussed on presenting the results of a user-requirement analysis for AR technologies based upon focus group discussions with aviation SMEs. Because military and commercial aviation domains have pioneered HMD and AR technology for decades, the offered design insights from extensively experienced SME populations (in the form of military and commercial pilots) is highly profitable to developing general AR design guidance, as well as aviation specific guidance. A thematic analysis grounded in the three-level SA model was employed to generate high-level situation awareness design requirements for a commercial aviation HMD and for its associated AR symbology.

2 Materials and methods

2.1 Participants

Eleven aviation SMEs, all airline transport pilot licence (ATPL) holders (mean/SD years flight experiences = 23.3/9.0), participated in the study. Participants had a range of aircraft experience with 6 participants having HUD/HMD experience (Table 1). The sample size for the study was determined using a qualitative saturation technique, whereby recruitment ceased once the data no longer generated novel design requirement insights. The study received ethical approval from Coventry University.

Table 1 Summary of aviation subject matter experts
Full size table

2.2 Data collection

Participants joined one of three different focus group sessions. These were hosted via Microsoft Teams, due to the participants being geographically dispersed and pandemic ‘social distancing’ guidelines in place at the time of data collection. Focus groups were conducted in a semi-structured fashion where an informal conversational tone was utilised. To provoke participant requirement elicitation, two broad discussion topics were presented on HMD Enhanced Operations and AR symbology which prompted discourse regarding the technologies’ potential performance and safety benefits. Focus groups lasted between 1.5 and 2-h.

2.3 Qualitative analysis

Focus group transcripts were analysed using TA in NVivo™ (version 1.5) to produce a 3-level hierarchy of design requirements. High-level primary themes, representing the division of HMD and AR design requirements, were declared in concert with the identification of emerging lower order applications and requirements for the technology (as secondary and tertiary themes) via deductive and inductive analytical approaches, respectively. Further deductive coding described how the technology requirements would enhance pilot situation awareness. This was achieved by linking the lowest order technology applications and requirements (the tertiary themes) to the three-level SA framework within which the user-requirement analysis was grounded.

Triangulation was used to address any coding inconsistencies. A separate researcher reviewed and recoded the data, resulting in an 83% agreement between original and recoded data. Subsequent dialogue between coders addressed these discrepancies until agreement was met.

3 Results

The organisation of secondary and tertiary requirement themes within the two inductively declared primary requirement themes, HMD Enhanced Operations and AR Symbology, is presented in Table 2. The faciliatory associations between tertiary requirement themes and the three-level model of SA framework are illustrated in Figs. 4 and 6.

Table 2 Hierarchical thematic structure of primary, secondary, and tertiary design requirements. Tertiary requirement descriptions included
Full size table
Fig. 4
figure 4

Hierarchical thematic map of the HMD Enhanced Operations primary theme. Facilitatory associations between tertiary design themes and the SA framework are shown at the bottom-level of map. Tertiary themes may facilitate more than one level of SA

Full size image

3.1 Theme 1: HMD Enhanced Operations

Overall, participants positively appraised HMD technology, with notable reference to how it would support SA in high task-load, low-level flight scenarios. Three secondary themes were explored here. Wider Field of Regard described operational tasks where participants viewed HMD applications would effectively improve SA. The secondary theme called HMD Usability represented participant requirements which would underpin the acceptance of HMDs by the pilot community. Optimise Information Processing specified how HMD designs must emphasise information processing optimisation to support SA

3.1.1 Theme 1a: wider field of regard

Participants frequently expressed the view that HMD applications would enhance operational performance and safety through the HMD’s capability to depict critical flight information across a wider field of regard. Operation specific SA enhancements of this capability are covered in the following 3 tertiary requirement themes: Complex Airspace Awareness, Attitude Awareness, and Head-Movement linked Symbology.

Complex Airspace Awareness was the main operational advantage of HMD technology that was identified, primarily due to the SA enhancement of peripheral information (Fig. 4) that HMD’s head and world-referenced symbology granted. The value of being able to perceive perspective traffic information in high workload low-level flight scenarios was emphasised:

[SME4] At least with the HMD, you can look out and say “that's where it is”. You know there's a symbol representing it. And again, it's a glance in those high pressure, high workload situations.

Similarly, the synthesis of perspective AR traffic information with currently available three-dimensional navigation information was raised as a likely benefit of HMDs in high workload traffic avoidance situations:

[SME3] If you get an airborne collision avoidance system (ACAS) warning or even just a ACAS traffic advisory, the first thing you instinctively want to do is try and visually acquire that aircraft. Trust me, if there's a lot of aircraft around, you won’t acquire the right one. So something like this would help.

The spatial SA benefits of HMDs extended to ground operations in low-visibility conditions. For example, SMEs highlighted how aircraft referenced AR symbology (e.g. the aircraft’s lateral undercarriage) could be paired with world-referenced AR symbology (e.g. proximal aircraft and airport traffic) to forecast their aircraft’s displacement relative to nearby airport hazards.

[SME9] The gold standard HMD guidance is where you could look over your shoulder and see the position of the main gear and see the position of the runway edge on an AR display.

[SME3] Navigating in the dark you’re just following the taxiway, but you don’t know what's around you. You're just hoping that everybody is doing the same thing.

The Attitude Awareness tertiary theme highlighted the advantage of head and world-referenced symbology in supporting attitude awareness and spatial orientation, namely through the addition of a “scene linked” AR horizon line (i.e. additional level 1 SA data), when paired with aircraft referenced power symbology. Participants held strong opinions that this capability would have significant safety benefits during upset recovery events:

[SME7] Some kind of HMD system would allow you to actually just add some sense, just give you situation awareness that, well, there’s the horizon, there’s the power. We can't possibly be stalling the aircraft.

Head-Movement Linked Symbology established the benefit that AR symbology would afford participants by enabling head-referenced ‘eyes-out’ symbology. In this case, the most significant operational benefit was the ability to maintain view of the primary flight instruments whilst conducting wider scan patterns of secondary head-down instruments.

[SME8] If that was following you around, and you were able to take that information with you while you were monitoring something else, I think that that's probably helpful.

3.1.2 Theme 1b: usability

The viability of a commercial aviation HMD will be largely determined by its user acceptance. SMEs underlined two important usability requirements: (1) the physical ergonomic properties of a commercial aviation HMD and (2) the necessary training requirements.

The HMD Physical Ergonomics tertiary theme exemplified how HMD acceptance will be largely driven by the level of encumberment that the device will entail. Many participants remarked that a lightweight HMD design would be a requirement for pilots. Particularly, if the HMD was to be worn for extended periods during flight (e.g. not just during critical departure and approach flight phases):

[SME11] Do you wanted to be wearing a helmet to fly for 10 hours on long-haul? Probably not. But you know, if you could have something like glasses then maybe that would be a solution. But I can’t see commercial partners wearing helmets.

Fortunately, these views echo previous requirement analyses (Arthur III et al. 2014; Tran et al. 2018) stipulating that a commercial HMD must consist of a lightweight design, as seen with recent experimental lightweight HMDs that are under development (Alvarez and Rodriguez, 2021).

Training Requirements for a commercial HMD system were remarked on by participants. Whilst training requirements are not about the design of AR symbology, per se, their consideration is relevant in the light that poor AR design will impact training requirements and cost and thus, organisational acceptance of AR technology. Unique perspectives on the feasibility of a commercial HMD were offered from participants with experience of HMDs from the military rotary domain. Whilst it is accepted that the mission parameters of military rotorcraft and fixed-wing commercial aviation operations are clearly different; there are aspects of the extensive HMD training demands from the military domain that will carry over to the commercial domain. For instance, the opportunity to overload HMD naive pilots with data:

[SME7] The amount of data that comes through that helmet mounted display, you know it can throw people. So there is an amount of training required to carry all of that data.

3.1.3 Theme 1c: optimise information processing

Comments from the SMEs with HUD and / or HMD experience produced HMD design requirements detailing the need for information processing optimisation. This was reflected in the proposal of requirements supporting level 1 and 2 SA (Fig. 4). In particular, participants proposed that a future HMD system must address pilots’ perceived frustration that the perception and comprehension of HUD information were compromised by the presence of task-irrelevant data within the HUD real estate. Furthermore, participants reported the information processing conflict caused by the deprivation of colour-coded information on monochrome HUDs. The associated HMD design guidance is organised into following tertiary themes: Maximise Task-Relevance and Colour Symbology.

Maximise Task-Relevance represented prominent remarks reflecting the significant range of tasks where HUD presented information is not relevant. In some instances, the lack of task-relevant symbology was reported to ultimately make the HUD an overall hindrance. A notable example being during ground operations where the majority of the HUD real estate remains dedicated to aviation related symbology:

[SME8] The head-up symbology doesn’t change for when we're on the ground, so it's giving us information we don’t need—because I don’t need to know when I'm on the ground whether I'm pitching up.

Further comments underlined the inadequacy of existing “decluttering” solutions that have been designed to partially address the issue.

[SME8] It still leaves a lot of information there which isn’t required for taxiing. So if we are in torrential rain, or anything like that, a lot of pilots just remove the heads-up display because it's just a hindrance at that point in terms of visibility.

Together the comments demonstrate the importance of developing symbology that go beyond the replication of traditional head-down displays within the head-up ‘eyes-out’ location. Specifically, the effectiveness of future HMD symbology will be defined by displays with the capability to support pilot attain level 2 and 3 SA through the provision of time critical task-relevant information.

Colour Symbology linked to the monochrome display format of HUDs that participants cited as an information processing issue. In display design colour can bestow a pre-attentive processing advantage to task critical visual information (Neisser 1964). In this way, colour coding will improve both level 1 and 2 SA. Indeed, pilot performance and workload are superior when colour-coded HUD symbology is provided (Blundell, Scott, et al. 2020a, 2020b). Ordinarily the lack of colour on a HUD is compensated for in HUD design by bolstering the salience of task critical information using other visual parameters, such as increasing the relative size of the task-relevant symbology. Despite these compensations, participants highlighted a requirement for colour-coded symbology on a future HMD. Figure 5 exemplifies this reported colour salience issue by depicting a side-by-side comparison of an urgent wind shear warning within a colour PFD and monochrome HUD.

[SME8] All of those warnings are bright red on the PFD. But on the HUD it's just green, you know, so it doesn’t have the same effect.

[SME10] The HUD being monochrome, you think goodness me it’s all the same. How do you choose the wheat from the chaff?

Fig. 5
figure 5

Information salience comparison of colour and monochrome wind shear warning information presented on a Boeing 737 PFD a versus on a generic HUD b, respectively

Full size image

3.2 Theme 2: AR symbology

The other primary requirement theme was AR Symbology. Two secondary themes were explored here: (1) An overview of possible AR applications, and (2) the mitigation of potential risks associated with presenting information in an AR modality.

3.2.1 Theme 2a: applications

Participants described three AR applications that they believed would support pilot SA and improve operational efficiency and safety. These included proposals that would improve the perception and awareness of airspace properties, traffic, and cockpit controls and displays. Early research on several of these proposed applications has been conducted (Li et al. 2020; Tran et al. 2018). Two other proposals, terrain and flight path AR information, have already been developed and deployed in commercial or military rotary and fixed-wing operations (Blundell, Huddlestone, et al. 2020; Stanton et al. 2019). Despite this, participants described novel implementations of these applications that would bring operational improvements.

In terms of situation awareness, airspace visualisation and traffic presentation were two applications that have clear potential in directly supporting the acquisition of level 2 SA information, instead of requiring pilots to acquire and synthesise level 1 SA data from disparate sources (Fig. 6). Likewise, flight path information (such as a HITS) offers the pilot perspective information in a manner that directly supports level 3 SA information attainment. Previous research has demonstrated that directly presenting level 2 or 3 information to operators effectively bypasses lower SA processing levels (1 and/or 2) (Endsley and Jones 2012), which consequentially reduces operator workload (Blundell, Scott, et al. 2020b).

Fig. 6
figure 6

Hierarchical thematic map of the AR Symbology primary theme. Facilitatory associations between tertiary design themes and the SA framework are shown at the bottom-level of map. Tertiary themes may facilitate more than one level of SA

Full size image

Whilst participants highlighted how a HMD’s wider field of regard would support complex airspace awareness (see Theme 1), the Airspace Visualisation tertiary theme reflects how HMD AR imagery could be leveraged to visualise certain airspace hazards. In particular, visualisation of weather and other restricted airspaces (e.g. no-fly zones) were highlighted as advantageous applications. Regarding weather, recurring comments related to the difficulty of detecting and navigating around embedded cumulonimbus clouds.

[SME3] Stuff like an embedded cumulonimbus that are quite difficult to acquire visually. Uhm yeah, we do have the weather radar, but of course they are subject to shadowing effects where you can’t see on the radar where the actual embedded cumulonimbus are.

Opinions were offered on the use of AR imagery to depict volumes of restricted airspace, often referred to as threat envelopes or treat ‘umbrellas’ in the military domain (Robinson 2018), to support strategic planning around hazardous weather and restricted airspace.

[SME3] Sometimes you tend to go off on a tangent from your course because you're avoiding weather. It’s very difficult sometimes to see if you’re heading towards the danger area or a prohibited area just because you went off the airway, so if you can have that transposed in front of you it would definitely help.

Supportive findings for perspective 3D synthetic vision displays have recently been reported (Boyer et al. 2016; Olmos et al. 2009). However, whilst a perspective weather display is appealing, recent research (Wickens and Ward 2017) has shown that presenting traffic and weather hazard information within a 3D perspective display generates more ambiguity compared to when the same information is presented on 2D coplanar displays. These studies investigated 3D perspective symbology exclusively with fixed view head-down displays. Hence, the effectiveness of similar head-up AR symbology, such as on an HMD permitting a wide field of regard, is yet to be undertaken.

The Traffic tertiary theme was similar to Airspace Visualisation since many participants commented on how AR traffic information would complement a HMDs wider field of regard advantage. In some cases, these opinions related to having a relatively simple HMD replication of head-down traffic information (e.g. airborne collision avoidance system (ACAS) information). Though, the spatial ambiguity criticisms of weather displays are likely to be also relevant for traffic AR symbology displays (Alexander et al. 2009; Wickens 2000).

[SME9] It is stuff that you can build from a simple 2D ACAS display, but it’s not always intuitive and to be perfectly honest, the closest calls I’ve had are at those airfields where the standards of controlling and separation have been degraded.

The capability to highlight Critical controls and Displays within the cockpit was a desired AR application of the SMEs. Notably, the participants stressed the importance of this application when cockpit visibility conditions are compromised (e.g. a cockpit fire). The application would have clear safety benefits due to supporting level 1 SA perception when natural visual information has become degraded:

[SME4] Having something that you could put on to help you see in a low-vis environment, something that you could just put on along with your oxygen mask and say right there are the power levers.

Participant views emphasis the value of recent experimental findings (Li et al. 2020; Tran et al. 2018) that have demonstrated the performance and workload benefits of using AR is this way.

The importance of having symbology that provided projected Flight Path information was expressed, notably, during non-nominal high-task load situations (e.g. low visibility, high wind flying conditions):

[SME11] What would be incredibly helpful, you know where you’re working really hard as a pilot to navigate the aircraft safely, is some kind of visual guidance. Almost like flying through a tube, or some kind of visual projection of where you're going.

The speculated benefits of AR navigation information were not limited to flying tasks. Similar to the proposal that the HMD’s wider field of regard would support tactical spatial SA during taxiing tasks, strategic taxi navigation in low-visibility conditions could also be enhanced by AR information:

[SME5] There would be great benefit to be able to come up with, you know, sort of overlay of elements, perhaps of navigation display (ND) information on the PFD so you know, sort of like turn left or exit runway that’s over there.

Support for this proposal comes from research conducted by NASA between 2007 and 2014. A key finding from the project was that AR taxi navigation symbology, presented on either a HUD or HMD, led to superior taxiing performance and situation awareness, compared to head-down navigation aids (Arthur III et al. 2014). More recent research by Blundell et al. (2023) has corroborated these NASA findings by demonstrating the performance and workload benefit of conformal AR taxi navigation information that is paired with redundant haptic information.

Depiction of Terrain information was perceived to provide clear safety benefits. As would be expected, instances where the presence of AR terrain information would be most valuable was during low-visibility, low-level flight operations. Consequently, terrain AR symbology can be regarded as ‘restoring’ the quality of the natural degradation of the terrain’s level 1 SA perceptual visual data during these circumstances:

[SME1] Terrain [information] would be something that would be very interesting to have.

[SME11] Energy management won’t kill you; it might make you go around, but the terrain will definitely kill you. So having that mapped onto some kind of visual mapping would be what people would want out of some kind of display.

3.2.2 Theme 2b: mitigated risks

The Mitigate Risks secondary theme represented the SME requirements that mitigated the potential risks associated with making AR commonplace on commercial cockpits. Comments largely reflected the principles of good design and common criticisms of poor display design (Melzer et al. 2009; Federal Aviation Administration AC 25-11B, 2014). This included minimising the opportunity for display clutter and avoiding potential complexity creep.

Minimise Display Clutter linked to key principles of HUD and HMD symbology design that all SMEs were cognisant of. Clutter is considered to be the extent that AR symbology overlays and masks critical external visual scene information (Ververs and Wickens 1998; Yeh et al. 2003) and is regarded as a major threat to operator level 1 and 2 SA (Fig. 6). Participants’ appreciation of this threat was covered in several discussions, as demonstrated in the following extract:

[SME9] Just gonna say with all of this in terms of clutter. If this system, whatever it is, could have filters you could be the pilot and select it on or off depending on whether they needed that information.

Avoid Complexity Creep echoed participant concerns of the possible proliferation of task-irrelevant AR symbology. This is a common criticism of more technology-centric design approaches (Endsley and Jones 2012), whereby the overall system usability is undermined by designers unwittingly unleashing complexity into a system through the practice of feature escalation. The following SME comments captured pilots’ awareness of this issue in the context of overloading pilots with irrelevant traffic information on an HMD:

[SME6] This issue of complexity versus all the functionality, it’s this age-old automation problem, isn’t it? We want lots of functions, but we want it to be simple.

[SME2] I think a lot of those sorts of examples would very much drip into the data overload. It might be useful for me to fly around the London TMA in my Airbus and to know that that's an A320 from Barcelona. But actually, what am I gonna do with that information?

4 Discussion

HMD and AR technology will be a technological enabler of many future commercial flight concepts (Blundell, Huddlestone, et al. 2020; Moehle and Clauss 2015) and is an emerging technology in the medical, industrial engineering, and educational fields (Park et al. 2021). Despite this, human factors design guidance is lacking and has not kept pace with the technology’s technical advancements. A broad, high-level user-requirement analysis with experienced end-users, that provides design guidance for the development of a HMD and its associated AR symbology, has yet to be presented. To address this gap, a thematic analysis (Braun and Clarke 2006) using Endsley’s 3-level SA model as a framework was utilised to analyse SME focus group discussions involving potential commercial HMD applications. The analysis revealed operational areas where pilot SA weaknesses existed and generated design requirements where HMD technology and AR symbology could enhance SA.

The current study complements the growing body of evidence demonstrating the diversity of thematic analysis’ application within the field of design analysis (Agyekum et al. 2019; Babar et al. 2018; Bouamrane et al. 2019). Broadly, the analysis revealed that the implementation of HMD and AR technology on the commercial flight deck was welcomed by SMEs. Critically, operational areas where monitoring of peripheral information is required were identified as areas where HMD would be most advantageous. In addition, a range of novel AR applications were proposed that the SMEs believed would aid operator SA, including navigational (e.g. HITS) and terrain information. More novel proposals were offered in the form of using AR to highlight critical displays and instruments and to depict characteristics of the surrounding airspace environment (e.g. airspace restrictions).

Proposed HMD applications capitalised on the technology’s wider field of regard capability to enhance pilot perception (level 1 SA) of peripheral information. Indeed, several SMEs stressed the potential value of AR in providing clearer spatial orientation information to operators who found themselves in unusual attitudes, namely through the depiction of a scene linked horizon line. Subsequent upset recovery proposals were offered in the form of an AR HITS which projected a recovery flight path that minimised stress on the aircraft (level 3 SA). In the past two decades, 25% of all fatal aviation accidents were attributed to Loss of Control-Inflight (Boeing 2017), often involving unusual attitudes. Hence, a clear safety application of an AR solution is possible for these situations. Similar level 1 SA advantages were identified which involved AR symbology linked to the pilot’s head movements. Two high-task load applications of this capability were offered. The first was the ability of AR to allow the perception of primary instruments to be maintained during secondary instrument scans or whilst supporting the co-pilot with critical tasks (e.g. system configuration). The other application of head linked symbology was the ability to highlight cockpit displays and instruments. Crucially, participants suggested the potential safety of this application during emergency events where visibility has become degraded, for example, when there is smoke in the cockpit.

Likewise, SMEs expressed a requirement that future AR solutions should focus on presenting task-relevant information as a means to bolster the processing of level 2 and 3 SA information. Potential means to achieve this included the implementation of colour-coded symbology and for symbology to be more innovative, rather than being simply replicative of existing head-down display information (as seen on current commercial HUDs). In terms of SMEs’ desire for a greater variety of task-relevant symbology, the comments underline the pivotal role that user-centred design approaches, such as the user-requirements analysis presented here, have in the design and development of future displays intended to support operational safety and efficiency within complex environments such as aviation (Parnell et al. 2021).

These potential AR applications reinforced the discontent voiced by SMEs regarding the current lack of task-relevant symbology. Using Endsley’s SA model as the basis to understand cognitive requirements provided a novel approach to describing the design requirement for AR solutions. Indeed, from an SA perspective, the majority of proposed AR applications directly supported level 2 and 3 SA and minimised the relatively cognitive intensive requirement involved in sourcing and synergising lower-level data (level 1 SA). Previous research has demonstrated that presenting level 2 or 3 information to operators effectively bypasses lower SA processing level (1 and/or 2) (Endsley and Jones 2012), which consequentially reduces workload (Blundell Scott et al. 2020b). Ultimately, SMEs’ desire for more task-relevant symbology can be viewed as a requirement to strive for display designs that are more intuitive and which minimise the opportunity for overloading the user with irrelevant “data”.

5 Conclusion

Early involvement of human factors within the design process of emerging technologies is important to generate design outcomes that are usable and safe for current and future complex, and safety critical, activities. The current study demonstrates how principles from Endsley’s three-level model of SA can be applied within a user-requirements analysis to generate high-level design requirements for HMD and AR applications intended for complex operational environments (e.g. aviation, medicine, automotive). By involving experienced aviation SMEs, detailed analysis of focus group discussions identified the design requirements for HMD technology and AR symbology that could improve operational safety and efficiency. However, SMEs also raised concerns about the potential for system complexity creep and the opportunity for information overload that a commercial AR HMD system could introduce. This underlines the importance of conducting future, focussed, user-centred design studies to build upon and evaluate several of the proposed HMD applications before prototype development, testing, and assessment of them is undertaken. UCD practitioners and researchers, aiming to build upon and/or validate the current SA requirements of AR, are encouraged to implement standardised usability measurement tools (e.g. Heuristics evaluation (Molich and Nielsen 1990), System Usability Scale (Brooke 1996)) to evaluate future AR designs that have been informed by Endsley’s three-level model of SA.