Using augmented reality to reduce workload in offshore environments

In the offshore environment helicopters are widely used to transport crew and material from and to maritime wind farms. Due to unforeseeable and often inclement weather situations and challenging tasks these missions put a high workload on the helicopter pilots. In this paper two test campaigns are described which assess the utility of an affordable commercial-off-the-shelf (COTS) head-mounted display (HMD) to reduce workload for commercial maritime operations. The HMD system was implemented within the air vehicle simulator (AVES) at the German Aerospace Center (DLR). Three missions were flown with experienced offshore pilots, performed in a realistic scenario. Independent subjective assessments of both workload and situational awareness were obtained. Results from the studies show that the overall workload for all missions decreased and situational awareness increased when using the HMD. Opinions regarding overall benefit and advantages of the system were found to vary between pilots and missions.


Introduction
The increasing number of offshore wind farms has led to the continued growth of the maritime commercial helicopter market. Helicopters are used for the majority of transport and observation missions due to the speed and operational capabilities. In comparison to missions performed onshore, the offshore environment demands higher helicopter safety standards and crew experience requirements. Often when flying offshore, inclement weather conditions (e.g., precipitation, cloud) can lead to low visibility, subsequently leading to Degraded Visual Environments (DVE). Most wind farms in close proximity to the German coast are located in the uncontrolled airspace class G. The minimum visibility for helicopter operations under visual flight rules (VFR) in this airspace is 800 m [3].
If the visibility is lower, missions must either be aborted or cancelled. Figure 1 illustrates an approach at the offshore substation EDYV in the windfarm Alpha Ventus with a low cloud base and low visibility. Flights in such low visibility with an absence of sufficient visual references may lead to a reduction in handling qualities (HQ) [1], to a higher workload [9] and to a reduction in situational awareness with a significant risk of spatial disorientation. All those factors can compromise flight safety.

3
During maritime missions, both the pilot and the helicopter typically operate close to performance limits. Offshore helidecks on wind farm substations, ships or research platforms often provide no more than the minimum ICAO recommended landing site size (i.e., twice the overall length of the largest allowed helicopter) [8]. Picking up or releasing personnel requires a stable hover next to a wind turbine or a ship. This is a challenging task, since it requires high concentration to maintain the desired position of the helicopter within potentially turbulent air. Furthermore, this hover out of ground effect (HOGE) is often performed with the engines operating close to the respective performance limits. As a result, the pilot must closely observe the head-down engine instruments, to continuously monitor the component performance parameters. In dual-pilot operations, as performed by most civil operators as well as organizations, such as the Flight Service of the German Federal Police, those tasks require good crew coordination. In single-pilot operations, as performed by a limited number of commercial offshore operators, these tasks cause a high workload, especially during adverse weather situations.
To increase situational awareness in DVE conditions, additional cues can be provided. It is well known that headmounted displays (HMD) can be used to supplement other cues available to the pilot. A reduction in workload has been demonstrated in previous studies (e.g., [12,13] and [15]). Currently, however, systems utilized in helicopters are expensive, and are, therefore, almost exclusively limited to military and research rotorcraft. Commercial offshore operators typically do not have the resources to install and maintain these systems.

Previous work
As part of the projects HELMA (Helicopter Flight Safety in Maritime Operations) and HEDELA (Helicopter Deck Landing Assistance), the German Aerospace Center has been developing and evaluating pilot assistance systems to reduce workload and increase safety when operating in offshore environments. In these projects, there is a specific focus on the use of affordable solutions, which could be used to support commercial and civilian operations. These projects have been undertaken in collaboration with the Flight Service of the German Federal Police, who is responsible for security and observation missions within German offshore wind farms.
After comparing various aviation-approved and COTS systems suitable as HMD in an research environment, the Microsoft HoloLens 1 was chosen due to its innovative technology, adaptability and affordability. It features binocular color displays with an approximate field of view of 30 • × 17.5 • , an inside-out tracking using filtered data from four cameras and an inertial measurement unit (IMU) as well as wireless connectivity via WiFi and Bluetooth.

Integration of the HoloLens in the AVES simulator
The HoloLens system was integrated into DLR's Air Vehicle Simulator (AVES), as exemplary shown in Fig. 2. This is achieved through a connection of the HMD application via WiFi to an access point, which is connected to the simulator network. An interface computer routes the helicopter state data, as calculated within the realtime helicopter flight model, to the HMD application. The application is built using the game engine Unity3D and is written in the script language C#. The objects displayed inside the HoloLens are referred to as holograms. As described in [14], there were a number of challenges encountered during the integration of the HoloLens system into the AVES facility.
One of these challenges was the alignment of the outside world presented on the simulator screen with the displayed objects inside the HoloLens. The dome projection of the simulator is a transformation of the generated world onto a sphere. This transformation needs to be repeated for the holograms in the HoloLens. To position the hologram sphere at the same position as the simulator dome sphere, a calibration process is necessary. A filter was integrated to remove "jittering" of head-fixed holograms. The image generation is set to guarantee a stable frame rate while providing detailed images without aliasing effects.
The inside-out headtracking of the HoloLens is driven by an internal filter algorithm which mainly uses the position and orientation data of an IMU for high-frequency movements and the image processing results of five cameras to determine its position within the environment. In various tests, it has been determined that the internal tracking is suitable within a stationary environment (i.e., a fixed-base simulation facility).
To use the system within a dynamic environment (e.g., moving platform, vehicle), additional compensation or external headtracking is required. This is currently being investigated in DLR as part of the HEDELA project, in preparation for planned flight tests. Due to the missing external headtracking only the fixed-base platform was used during the simulator trials.

Symbology in the helmet-mounted display
The HMD symbology is categorized into world-fixed, helicopter-fixed and head-fixed objects. While world-fixed objects are fixed to objects in the outside world, the helicopter-fixed objects are coincident with the position and orientation of the helicopter. Head-fixed objects always remain at the same position in the spectators field of view (FoV).
The head-fixed objects are based on the basic layout of a primary flight display without the horizontal situation indicator. As shown in Fig. 3, it consists of airspeed (1) and altitude (7) indicators, a bar displaying vertical speed (8) and a flight path marker (2).
The world-fixed objects are the heading indicator (3), an artificial horizon (6) and the highlighted obstacles (9). In the case of offshore investigations, obstacles include both offshore substations and wind turbines. The highlighted color can give additional information regarding the obstacle.
The attitude indicator (5) with a red slip indicator (4) is helicopter fixed, meaning its position is fixed to the helicopter nose.
When it is out of view for the pilot (by turning the head), a small auxiliary attitude indicator will be displayed in the center of the head-fixed objects.
While most aviation-approved HMD are monochromous green, the HoloLens uses full color RGB displays.
This can be used to supplement information relayed to pilots. Objects within the same category, e.g., standard PFD indicators can be assigned with the same color. Furthermore, the same color concept used on head-down displays can be transfered to the HMD. During the design of future systems, characteristics of human perception as well as environmental factors should be considered.
At the current design stage, it can be chosen between a full monochromatic (white or green) layout or a color coded design, as shown in Fig. 3.

Test environment
The simulation facility AVES is shown in Fig. 4. The simulator features three interchangeable modules; an Airbus A320 and an Eurocopter EC135 cockpit as well as a single aisle passenger cabin. These modules can be exchanged via a Roll-on-Roll-off (RoRo) system to utilize a full-sized six degree of freedom, hexapod motion platform or a fixed base platform. For the investigations described within this paper, the fixed base platform was used. The projection system in both platforms consist of 12 LED projectors each with a resolution of 1920x1200 which provide a horizontal FoV of 240 • and a vertical FoV of −55 • to40 • [2]. All hardware and software systems within the AVES can easily be modified, which qualifies the simulator for a broad spectrum of research activities. In the development process of new systems and applications the AVES is used as the test platform after a desktop simulation and before the flight testing using DLR's research helicopter ACT/FHS.
For the simulator trials, the EC135 cockpit was used. It is shown in Fig. 5 and has a similar instrumentation as the DLR's research helicopter ACT/FHS (Active Control Technology/Flying Helicopter Simulator). The helicopter flight dynamics were calculated using the HeliWorX real time model, based on the nonlinear helicopter model SIMH [5]. The HeliWorX model suite and its development is described in [4]. The model is based on the flight dynamics of the ACT/FHS.
To perform the simulator studies, a Rate Command (RC) system, with stability and control augmentation system (SCAS) was selected. It provides attitude control for the pitch and roll axis, a stability augmentation system (SAS) for the yaw axis and direct control for the collective axis. This type of control system corresponds with the type typically used for current offshore operations. A more detailed description of the vehicle handling qualities is contained within [9]. To enable the pilots to navigate through the wind farm, the integrated Garmin GNS-430 GPS was used. No higher automatic flight control system function (e.g., upper modes) were available.
Missions were undertaken in a realistic maritime scenario. This is based upon the wind farm Alpha Ventus, which lies about 25 NM north of the island Borkum in the German North Sea. The scenario includes twelve wind turbines, an offshore substation with a helideck designated as EDYV, as exemplary shown in Fig. 1, and the research platform Fino 1. To provide pilots realistic visual cues during landing, hover and hoisting maneuvers, the helideck of the offshore substation and the wind turbines including the hoisting areas are modeled in high detail. The scenery is visualized using an OpenSceneGraph based image generation.
A highly detailed wave generation and visualization is implemented using the Sundog Triton Ocean Software Development Kit (SDK). To improve the realism in the scenario, the direction and height of the waves as well as the orientation and rotational speed of the wind turbines are synchronized with the wind inputs used in the flight model.

Piloted simulator campaigns
This section details the missions and pilot experience from two campaigns, both conducted using the HMD and simulation facilities discussed in the preceding sections.

First simulator campaign
The first piloted evaluations of the developed symbology and maritime simulation environment were conducted in November 2017, as described in [15]. The focus of this study was to evaluate if the HMD can increase the flight safety and operational availability during offshore helicopter operations.
Five pilots participated in this study. Their flight experience is presented in Table 1. The average age of the pilots was 45.4 (SD 10.5) years. All of them were familiar with night vision goggles (NVG), two of them were in possession of an Instrument Flight Rules (IFR) rating. Two of the pilots regularly flew offshore missions. However, none of the pilots were familiar with the use of HMDs, here defined as visual assistance systems providing aircraft information to the pilots eyes. None of the pilots had an EASA CAT-1 or CAT-2 helicopter test pilot rating.
The test scenarios were developed together with pilots from the Flight Service of the German Federal Police. The scenarios consisted of critical elements of a typical mission;

Evaluation methods
During evaluations, the subjective NASA Task Load Index (TLX) method was used to determine workload. The NASA TLX [6] is a rating scale to evaluate workload in six dimensions on a scale ranging from 0 (workload very low) to 100 % (workload very high). The subcategories are Mental Demand, Physical Demand, Temporal Demand, Performance, Effort and Frustration. The ratings can be used as raw or weighted data. The weighting is performed by an additional workload contribution questionnaire for each of the six dimensions. To evaluate the situational awareness (SA) during the tasks with and without the HMD, the Situation Awareness Rating Technique (SART) was used. This was based upon the method using ten dimensions (10D) presented by Taylor [11].
The questions are grouped in the categories Demand (D), Supply (S) and Understanding (U) of the situation, each ranging on a seven point rating scale, as shown in Fig. 25. The final SART score was calculated by the formula SART = U − (D − S) . Pilots were asked to award NASA TLX and SART following the completion of each mission.

Mission 1: offshore search-and-rescue (SAR)
The first mission is a typical scenario performed by the Flight Service of the German Federal Police as federal coast guard organization and by helicopter operators in contract with wind farm operators. After receiving the order from a local rescue coordination center, the helicopter starts at its base, transitions into a cruise flight and descends to the target to perform the rescue maneuver. The simulated mission as shown in Fig. 6 starts in cruise flight at a position north-west of the island Borkum at 6500 ft ASL. The pilot is then instructed to proceed to waypoint HW751 and to descend after waypoint HW752 to the offshore substation in the Alpha Ventus wind farm. During the mission, the wind was 20 kt from 345 • with a broken cloud layer (cloud base at 1000 ft). The mission was completed after reaching the Alpha Ventus offshore substation.
An example of the vertical flight path recorded for Pilot 1 in Mission 1 is shown in Fig. 7. During flights without the HMD, most pilots chose to initiate an earlier descend to 100-300 ft at a distance of 3-8 NM to the platform designated as EDYV. With the HMD, pilots generally performed more continuous approaches, with a constant rate of descent ending at the platform. This indicates a potential increase in situational awareness.

Mission 2: navigation in the wind farm
The second mission was performed, as shown in Figs. 8 and  9, within the wind farm. The task starts at the wind turbine AV7 and continues in an"L"shape to the next turbine. The mission starts in good visual conditions (GVE) with a horizontal visibility of 3000 m. Upon reaching each target wind turbine, the visibility was reduced gradually in the steps 3000 m-700m-500m-400m-300m-100m. The pilots were instructed to abort the mission as soon as they considered the visibility too low for visual flight.
As already stated in the first analysis published in [15], using an HMD can bring multiple benefits compared to conventional displays. These are a reduced workload, increased situational awareness, perceived safety and operational availability for maritime environments in DVE.
The main objective of Mission 2 was to determine if an HMD has an effect on the minimum visibility acceptable for the pilots during maritime operations under visual flight rules (VFR). Table 2 shows the results of the pilots feedback on the visibility limit at which they would cancel the mission in real life. Here a clear difference was found between cases with and without HMD for three of the five pilots.
In the initial planning of the first campaign there was an additional hover mission which could not be conducted due to time constraints. Therefore, a second campaign was planned. Furthermore, during the first study, pilots asked for an engine indication in the HMD which could further decrease the necessity of glances to the head-down instruments.

Second piloted campaign
The focus of the second study was to evaluate a potential decrease of workload in a third mission with high-precision helicopter control requirements. It was also used as an opportunity to evaluate the improvements to the HMD that were conducted based upon feedback received during the first piloted simulation campaign. These improvements included an additional first limit indicator (FLI) and a drift indicator.
The initial symbology in the HMD as shown in Fig. 3 was extended for the second study. The FLI was placed under the airspeed indicator (1 in Fig. 3) and a drift indicator placed under the altitude indicator (7 in Fig. 3). The layout of both instruments is shown in Fig. 10. The FLI was configured to appear at values above 6.0, since the relevant engine limitations for the ACT/FHS are above this value.
In normal all-engine operative (AEO) mode, only the AOE relevant ranges and limits are displayed. When in one engine inoperative (OEI) operation, the additional limits, as noted in the respective rotorcraft flight manual, are displayed.
The drift indicator was configured to appear at a groundspeed of 10 kts. Below a groundspeed of 2 kts a higher zoom level is activated. The appearance and zoom values were determined by pilots feedback and the evaluation of mean groundspeed values during hover in previous campaigns.   During the second study, only pilots with offshore experience participated, as shown in Table 3. The average age of the pilots was 46.6 (SD 8.6) years. All of them were familiar with NVG goggles and two of them flew regularly with military helicopters using HMDs. Two of the pilots had an EASA CAT-1 or CAT-2 helicopter test pilot rating.

Evaluation methods
In addition to the NASA TLX and SART scales as introduced in the evaluation methods for the first campaign, the Bedford Workload Rating (BWR) scale was used for the evaluations in Mission 3. The Bedford Workload Rating scale is a ten point subjective decision tree based scale, used to evaluate pilot's spare capacity during a specific task [10]. The ratings range from "Workload insignificant (1)"to"Task abandoned. Pilot unable to apply sufficient support (10)", as shown in Fig. 24. A Lickert-type questionnaire as shown in Fig. 18 was used to query the pilots opinion on the FLI design.

Mission 3: hover close to a wind turbine
As with Mission 1 and 2, Mission 3 was conducted in the Alpha Ventus wind farm. It consisted, as shown in Fig. 11, of the final approach to the offshore substation to drop off personnel, a flight to the wind turbine AV4, a stable hover to pick up personnel by hoist and a return to the offshore substation. After the first landing at the offshore substation, the weight of the helicopter was reduced by 100 kg, to simulate a drop off of personnel. During a power check while hoisting at the wind turbine the weight of the helicopter was increased by 100 kg.
The pilots were instructed to comply with the 30 s OEI limit (128 % torque for one engine or 64 % with both engines) during the hosting at the wind turbine. During the landing maneuver at the offshore helideck and during hosting, pilots were asked to follow standard offshore operation rules (e.g., CAT A landings / take-off).
As in real operations, the head and blades of the wind turbine in the visualization are rotated pointing into the wind direction during this mission.

Results
The objective analysis of the recorded mean true airspeed (TAS) and flown distance during Mission 3 are shown in Tables 4 and 5. With the exception of Pilot C, only minor differences were found between flown distances from the platform to the wind turbine and mean TAS. The flight path of Pilot C is shown in Fig. 12. While the other pilots flew almost the same flight path with and without the HMD, Pilot C chose a much shorter path with HMD. This indicates a higher situational awareness with the HMD for Pilot C.
Consistently through the flights, all pilots kept a safer distance to the wind turbines with the HMD. An example is shown for Pilot A in Fig. 13. This indicates that the situational awareness is increased by the supplementary information displayed in the HMD.
Cyclic deflections from Mission 3 tests are shown in Fig. 14. Pilots A and C show a different control behaviour with lower amplitudes in the longitudinal and lateral axis to stabilize the aircraft with the HMD.   These pilots were the most experienced involved in the study. This result suggests that experienced pilots were better able to utilize and process the additional information provided by the HMD to stabilize the helicopter.

Situational awareness
In both campaigns an increase of situational awareness, with the exception of Pilot 2 in Mission 1, can be seen from the SART evaluation, as shown in Fig. 15. In Missions 2 and 3, where the pilots had to fly in the wind farm close to the wind turbines at DVE conditions a similar increase of the SART score ( ΔSART 1,2 = 9 ) can be seen. This corresponds with pilot feedback and comments.
For all three missions, the situational awareness rating with the HMD are less scattered. This result might be an indication that the information displayed in the HMD reduces the impact of pilot experience on situational awareness.

Workload
The raw NASA TLX overall workload ratings for all missions are shown in Fig. 16  A higher overall workload with the HMD was experienced by one pilot in Mission 1 and by two pilots in Mission 2.
The complete NASA TLX scores as shown in Figs. 20, 21, 22 show a wide spread in all subcategories with and without the HMD for the individual pilots. Since the workload of a task is strongly dependent on the experience, this is an expected result considering the differences in pilot experience as listed in Tables 1 and 3. During the first campaign (Mission 1 and 2) all of the pilots experienced a lower or equal physical demand and four of five pilots a higher or equal temporal demand with the HMD.
During the second campaign, all pilots agreed on a lower physical demand and a better performance during the task using the HMD. In comparison to Mission 1 and 2 the overall workload rating in this mission is consistently lower with the HMD for all pilots.
Judged by the recorded flight data as noted in Tables 4  and 5 there is only a slight increase of mean velocity and a decrease of the flown distance for three of five pilots.
The Bedford Workload ratings for Mission 3 (hover task) are shown in Fig. 17. All pilots gave either the same or a better workload rating for the run with the HMD.

Symbology
All pilots asked for a decluttering function with different information layouts, e.g., for cruise flight and hover. They liked indicators which have the same dimension as the headdown indicators. In addition to the general helicopter performance limits four of five pilots asked for an indication of the operational limits.
These are limits required for CAT A and performance class 3 operation and for passenger comfort. The implemented FLI engine limits for performance class 3 were considered extremely helpful during hover operation by all pilots. As stated by two pilots the drift indicator misled to an excessive "follow-the-needle" precision. It was considered useful in environments with few visual cues, such as hoisting on open sea. The result of a Lickert questionnaire with questions related to the circular FLI layout is shown in Fig. 18. All pilots agreed that a FLI in an HMD can decrease the workload and that they would use it in good as well as bad weather situations.
Additional to the questionnaire, three of five pilots commented that the FLI/torque value below the indicator is an unnecessary information if the performance margins are color coded. Two pilots appreciated the additional numerical display to be aware of the exact remaining performance. All pilots preferred the circular FLI design over a line or bar layout, as shown in Fig. 19, since the perception of qualitative information and the rate of change was judged to be faster especially in tasks with high workload. The feedback on the color concept of the display shows that pilots preferred the color coded design approach over a monochrome green or white layout, since this enables a filtering between the displayed information. All pilots would prefer a similar color concept in the HMD as in the head-down displays. Three pilots would reserve the color red for critical values on indicators and in a critical vicinity or on a critical path to an obstacle.

Discussion
To determine the significance of the presented results, nonparametric statistics were applied. These are used with small sample size data sets, which are not normally distributed. For the evaluations with the two observations without and with HMD, a sign test was conducted. The result of the subjective data presented above shows that none of the results is significant within p = 0.05 . This can be traced back to two main factors: -Only a very small sample size (five subjects) was used.
-The change in parameters (deltas) of the values with and without HMD differs substantially between the individual pilots.
The results might also be influenced by familiarization effects with the environment. After having flown a couple of times within the scenery, in this study the wind farm Alpha Ventus, pilots have a higher situational awareness, recalling the position of obstacles. Furthermore, during the first campaign, three of five participating pilots had no experience flying in wind farms. In the second campaign all pilots had experience flying in wind farms (mostly other than Alpha Ventus) before. Therefore, a comparison of results collected between the two campaigns is difficult, especially with regard to situational awareness.
The task performance requirements (e.g., following a flight path) were deliberately not stringently constrained, to evaluate the difference in behavior without and with the HMD. This, however, complicated the comparison of objective data between the pilots, since they followed different procedures. As an example, the pilots performed their usual CAT A take-off and landings with different vertical speeds ranging from 300 to 500 ft/min as defined by their company (e.g., for comfort). A number of pilots complained about the weighting questionnaire for the NASA TLX as they were unable to prioritize the workload dimensions over each other. As a result only the raw TLX results were used for the assessment.
During the campaigns some pilots mentioned ergonomic issues with the HoloLens 1 glasses as HMD. They perceived the reduced transparency of the glasses as annoying especially in situations with low external light. To clarify the influence of this issue on objective and subjective metrics further investigation is necessary.

Conclusions
The focus of this paper was to evaluate a potential increase in situational awareness and a decrease in pilot workload when performing offshore operations utilizing an affordable HMD system. The AR glasses Microsoft HoloLens was selected for the investigation, and subsequently implemented in the AVES research simulation facility. During two simulator studies, potential benefits of the system were examined. In the first study, an offshore search-and-rescue and a navigation scenario was flown by the pilots.
As a result of pilot feedback from the first study, additional elements were added to the displayed symbology in the HMD. These were assessed during a second study. This consisted of a hover mission at a wind turbine.
From the analysis of the two simulator studies the following conclusions can be drawn.  potential benefits in workload or situational awareness can be implemented and tested with a lower effort than with commercial aviation-approved HMD. 2. All participating pilots concluded that they would welcome an HMD system like the one used in this study for use during commercial or police helicopter operations. 3. The integration of engine monitoring indications is considered essential during operations close to ground like hover, landing and take off by all pilots. While in a two pilot cockpit configuration this is an optional information for the pilot flying, it was considered essential in a single pilot configuration. Especially in commercial offshore operations, pilots are required to perform flights in the highest performance class to ensure passenger safety. To guarantee being in the correct performance margins, pilots need the engine indication in high workload maneuvers (e.g., hovering while deploying personnel). The layout of the engine indication needs to be adjustable to the respective helicopter model avionic layout and to pilots habits. Nearly all pilots preferred a layout in the HMD which is close to the head-down indication of their usual helicopter model. 4. Since the pilots had outside visual cues during the hover maneuver at the wind turbine, most pilots considered the drift indicator unnecessary. Only during situations with poor or no usable visual cues (e.g., like hoisting in open sea) this system might increase performance and reduce workload. This scenario could be tested in future research efforts. 5. During the Missions 1-3, no significant improvement in the objective metrics during the flights with the HMD can be seen. Evaluating the data from the experimental flight data recording system, variables such as the distance flown, time between mission segments, speed or control deflections are not significant different without and with the HMD. 6. During the Missions 1-3, an improvement for the subjective metrics can be seen for all pilots either in an improved situational awareness or a decreased workload.
7. The situational awareness, as evaluated using a SART 10D, increases for nine out of ten pilots using an HMD in Missions 1-3. 8. The workload, as evaluated using the Bedford Workload Rating, decreases for three pilots and remains constant for two pilots using the HMD in Mission 3. 9. The overall workload, as evaluated using the NASA TLX score, decreases for 12 out of 15 evaluations using the HMD in Missions 1-3.