Keywords

1 Introduction

In recent years, the field of aerial service robotics applications has seen a rapidly growing interest in the development of Unmanned Aerial Vehicles (UAVs) equip-ped with some autonomous capabilities. However, since UAVs often operate in high uncertainty and dynamic scenarios characterized by unpredictable failures and parameter disturbances, no totally-autonomous control system has emerged yet [1]. Supervisory systems including human in the control loop are required to both monitor UAV operations and assist UAV controllers when critical situations occur [2, 3].

Systems equipped with flexible levels of autonomy (LOAs) and decision-making capabilities in uncertain environments may be exploited to dynamically allocate human-machine functions by discriminating situations where operators’ skills are sufficient to perform a given task from situations where system suggestions or interventions may be required [4,5,6]. The assessment of operator multitasking performance as well as the level of his/her mental effort for monitoring UAVs, generally termed as “cognitive or mental workload” [7], may be used to determine which LOA is needed for the system.

By leveraging the above considerations, this paper reports on the activities that have been carried out at Politecnico di Torino and at TIM JOL Connected Robotics Applications LaB (CRAB) to develop, through an assessment of humans’ mental workload, an adjustable autonomy system equipped with some decision-making capabilities in UAV-traffic monitoring scenarios. The system, later referred to as “control tower”, was devised to autonomously infer the appropriate level of autonomy by exploiting a mental workload prediction model built on operators’ cognitive demand in monitoring a growing number of UAVs with an increasing level of risk.

A simulation framework was developed to reproduce both swarm of autonomous drones flying in a 3D virtual urban environment and critical conditions they could be involved into. Afterwards, a user interface showing the 2D map of the city was developed to both display drones’ positions and drones’ flight information and allow human operators to monitor and intervene when critical conditions occur. A Bayesian Network (BN) classifier was exploited in this work to build the mental workload prediction model described above. This classifier was also leveraged as learning probabilistic model due to its capability to solve decision problems under uncertainty [8].

A user study was carried out with several volunteers, who were asked to perform some supervision and monitoring tasks of a variable number of drones with a growing level of risk. During each experiment, participants were asked to evaluate their perceived mental workload in order to train the system developed inferring the appropriate level of autonomy accordingly.

The rest of the paper is organized as follows. In Sect. 2, relevant literature in the area of adaptive autonomy systems is reviewed. In Sect. 3, the architecture of the system proposed in this study is described. Section 4 provides an overview of the user interface exploited in this study. Section 5 introduces the methodology that has been adopted to perform the experimental tests and discusses results obtained. Lastly, Sect. 6 concludes the paper by providing possible directions for future research activities in this field.

2 Related Work

Many studies in aerial robot applications domain have investigated the evaluation and classification of cockpit operator’s workload.

A number of studies have revealed the advantages in exploiting dynamic function allocations for managing operator workload and maintaining him or her focused in control loops [9, 10]. In literature, several criteria have been investigated to evaluate human’s cognitive load. The main measurement techniques have been historically classified into three categories: physiological, subjective, and performance-based [11]. Different techniques for mental workload assessment and classification have been proposed in this field.

Many research studies have focused on physiological measurements for assessing operator cognitive load in real time. For instance, Scerbo et al. [12] proposed the EEG power band ratios as example of workload measurement in adaptive automation. Wilson et al. [13] exploited EEG channels, electrocardiographic (ECG), electrooculographic (EOG), and respiration inputs as cognitive workload evaluation and an Artificial Neural Network (ANN) as classification methodology. Magnusson [14] examined the pilots’ Heart Rate (HR), Heart Rate Variability (HRV), and eye movements in simulator and real flight.

Despite these studies have provided evidences in merging more than one physiological measurements to improve the accuracy of workload classification [13, 15], such approaches have proved to be very infeasible from a measurement perspective, affected by the emotional state of the operator and impractical in aircraft cockpits application due to the need of wearing different devices at the same time [16].

In parallel to these studies, other approaches were investigated involving physiological measures in combination with other classes of workload assessment techniques. As a matter of examples, in [8] the authors performed operator’s workload evaluation in piloting a flying aircraft by using EEG signal with NASA-TLX questionnaire as subjective measure and a Bayesian Network as classification method. Di Nocera et al. in [17] have investigate operator’s workload evaluation engaged in simulated flight employing the eye fixations measure and NASA-TLX questionnaire as assessment methodology and Nearest Neighbor algorithm (NN) as classification method. In [16], the authors investigated different classes of cognitive workload measures by merging cardiovascular activity and secondary task performance (a performance-based technique), as inputs to an Artificial Neural Network (ANN) for operator cognitive state classification during a simulated air traffic control task.

Based on the short but representative review above, it can be observed that the panorama of mental workload assessment and classification techniques in aerial robotics applications is quite heterogeneous. By taking into account advantages and drawbacks of the above solutions, the system proposed in this paper combines subjective workload assessment techniques with a probabilistic Bayesian Network classifier to support UAV controllers in monitoring operations by autonomously inferring the appropriate LOA for the specific situation.

3 Proposed System

In the following, the adjustable autonomy system will be introduced, by providing also some implementation details.

3.1 Architecture Overview

The Adjustable Autonomy System Architecture (AASA) implementing the basic idea inspiring the present paper is illustrated in Fig. 1. It consists of three main components: UAVs Simulator (left), Bandwidth Simulator (right) and Adjustable Autonomy Control Tower (down). More specifically, the UAVs Simulator is the block devoted to load the 3D urban environment and execute the 3D drones flight simulation in it. A 3D physics engine was also exploited to test different flying scenarios in conditions as similar as possible to a realistic environment. The Bandwidth Simulator block was used to reproduce the network transmission rate of the simulated city. Since drones communicate or send information through the network, a low bandwidth connection could lead to critical conditions for UAV controllers. The Adjustable Autonomy Control Tower hosts Alert and Decision modules. The former determines the state for each drone by mapping the set of information collected by UAVs and Bandwidth Simulators, i.e., drones’ battery level, their distance from obstacles, with different levels of risk, later referred to as “Alert”. Three different levels are used to discriminate the drone’s level of risk, namely: “Safe”, “Warning” and “Danger”. The latter is responsible for establishing the appropriate level of autonomy by elaborating both the operator’s mental workload and his performances via the “Alert” level of each drone.

Fig. 1.
figure 1

Adjustable autonomy system architecture.

Full size image

3.2 UAVs Simulator

The UAVs Simulator is the module responsible for performing the 3D drones’ simulation in an urban environment. It consists of three different modules namely Autopilot, Physics Simulation and Ground Control Station (GCS).

The Autopilot module contains the flight software allowing drones to fly stable during the flight. More specifically, the Software-In-The-Loop (SITL)Footnote 1 simulator was exploited to run the UAV flight code without any specific hardware. Within this simulation tool, the un-compiled autopilot code, which normally runs on the drone’s onboard computer, is compiled, simulated and run by the SITL simulation software itself. In the specific case, the SITL software was used to run the PX4 Autopilot FlightcodeFootnote 2, an open source UAV firmware of a wide range of vehicle types.

The Physics Simulation module is responsible for replicating the real world physics of drones’ flight. In this work, GazeboFootnote 3 was exploited as a real-time physics engine in order to emulate the 3D models of UAVs, their physic properties and constraints and their sensors (e.g. laser, camera) in a 3D urban environment. Gazebo runs on Robot Operating System (ROS)Footnote 4, which is a software framework developed for performing robotics tasks.

The Ground Control Station (GCS) module contains the software needed to setup drones’ starting GPS locations, get real-time flight information, plan and execute drones’ missions. The communication between the PX4 Autopilot Flightcode and the GCS module is provided by the Micro Air Vehicle ROS (MAVROS) node with the MAVLink communication protocol. As illustrated in Fig. 1, MAVProxy node acts as an intermediary between the GCS and UAVs supporting MAVLink protocol.

Lastly, as illustrated in Fig. 1, this module provides UAVs information data to the Adjustable Autonomy Module by means of the RosBridge ProtocolFootnote 5. More specifically, these information regarding drones’ battery level, later abbreviated b and their distance from obstacles (e.g. buildings), later abbreviated o, are gathered from the Alert Module to determine the status of each drone.

3.3 Bandwidth Simulator

In this work, the network transmission rate was assumed to depend on two different variables: population density of the city sites (parks, stadiums, schools, etc.) and the network coverage. Three different values, in the range [1;3] - where 1 is “Low”, 2 is “Medium” and 3 is “High” - were used to describe the population density and network coverage levels of the city according to daily time slots and OpenSignalFootnote 6 data respectively. A grid on the map was created by storing in each cell the population density and coverage values described above in order to calculate the bandwidth in the considered area. The resulting transmission rate for each cell was computed according to a linear polynomial function y of the above values as follow:

$$Bandwidth=\left\{ \begin{matrix} High &{} \textit{if} &{} y< 0.5\\ Medium &{} \textit{if} &{} 0.5 \ge y < 1.5\\ Low &{} \textit{if} &{} y \ge 1.5 \end{matrix}\right. $$

As illustrated in Fig. 1, the three different calculated bandwidth levels (later abbreviated n) are sent to the Adjustable Autonomy Module in order to determine the transmission rate around the drone’s position on the map.

3.4 Adjustable Autonomy Control Tower

The Adjustable Autonomy Control Tower consists of two submodules namely: Alert Module and Decision Module.

The Alert Module, as illustrated in Fig. 1, receives data from the UAVs and Bandwidth Simulators as inputs. Each input is associated to three different variables, namely “High”, “Medium” and “Low” according to Table 1 and each variable is matched with a numeric value in the range [1; 3] - where 1 is “Low” and 3 is “High”.

Table 1. Drones’ information association to variables
Full size table

The mathematical formula described in (1) was exploited to compute the Alert:

$$\begin{aligned} y=\frac{1}{b-1}*\frac{1}{o-1}*\frac{1}{n-1} \quad \end{aligned}$$
(1)

where b, o, n, represent the three inputs listed in Table 1 and y represents the drone’s level of risk. Thus, the resulting Alert was calculated as follows:

$$Alert=\left\{ \begin{matrix} Danger &{} \textit{if} &{} b = 1 &{} \vee &{} o = 1 &{} \vee &{} n = 1\\ Warning &{} \textit{if} &{} 0.15< y < 1.5 \\ Safe &{} \textit{if} &{} y \ge 1.5 \end{matrix}\right. $$

It can be observed in (1) that when one of the input variables value is “Low”, the Alert assumes the “Danger” value. When the input variables values increase, then the Alert decreases from “Danger” to “Safe” through the “Warning” level.

The Decision Module represents the core of the devised architecture. It is responsible for inferring the appropriate level of autonomy by elaborating both operators’ mental workload and mission outcomes via the number of UAVs divided by “Alert” state.

A Bayesian Network (BN) classifier, which is a learning probabilistic model from data, was selected for representing both all variables involved in the study and their relationships in order to infer conclusions when some variables are observed. The structure of this model where the estimate LOA of the system is a direct child of the mission outcomes node via workload node is illustrated in Fig. 2. It was considered that the probability of changes in operators’ workload is conditioned on changes in the number of drones in “Alert” state. Thus, the probability to successfully complete missions is influenced by operators’ cognitive workload.

The LOAs proposed in this work, were namely: “Warning”, “Suggestion” and “Autonomous” where the system warns the operator if critical situations occur, suggests feasible actions to him or monitors and performs actions autonomously without any human intervention respectively.

Fig. 2.
figure 2

Bayesian Network model inferring the LOA from drones missions outcomes thus from subjective mental workload features via number of UAVs divided by “Alert” state.

Full size image

4 User Interface

In this section, a user interface showing the 2D map of the city for displaying drones’ positions and useful information for the human operator is presented. The devised interface allows the human operator to take control of drones through different flight commands. Depending on the current LOA of the system, the number or type of flight commands displayed dynamically changes thus defining the “Warning” or “Suggestion” interface.

Fig. 3.
figure 3

Warning interface (a), UAVs data summary (b), flight commands in Suggestion interface (c) and control and display information buttons (d). (Color figure online)

Full size image

A wide region of the operator’s display is covered with the 2D map of the city in which drones are shown in real time. A colored marker on the map is used to indicate both the drone’s GPS position and its current “Alert” (Fig. 3a). Three different color are used to depict the drone’s level of risk: green (“Safe”), yellow (“Warning”) and red (“Danger”). Drone’s marker color changes from green to red according to the linear interpolation described in (1). An extensive visual summary of data about each drone is shown on the panel in the right side of the interface (Fig. 3b). For each drone is reported its unique name, its battery level, the bandwidth coverage of the area around its location and its flying altitude. Right below the map are five controls buttons by which the operator can either issue flight commands or show information about the map or UAVs are placed (Fig. 3d). The “Start” button is used to run the UAVs simulation, whereas the “Options” button is used to show or hide the bandwidth coverage grid of the city and the drones’ paths. The other three buttons namely, “Land”, “Hovering”, and “Change_Path” are only available in the “Warning” interface and are used by the human operator to take direct control of the drone. In this modality, the UAV controller can land, hover or change the drone’s assigned path by defining the next waypoint with respect to the drone’s current position. On the contrary, in the “Suggestion” interface, the operator can only select actions among those suggested from the system in the summary panel on the right of the interface (Fig. 3c), according to Table 2. The replanning action implemented in this work provides an alternative path from the actual position of the drone to its target location by exploiting the Bing Map REST APIFootnote 7 with a route planning request.

Table 2. System suggested actions for each drone.
Full size table

5 Experimental Results

As anticipated, the goal of this paper is to build an adjustable autonomy system exploiting decision-making capabilities able to assist control tower operator by predicting mental workload changes or overload when the number of UAVs to be monitored highly increases. To this aim, a BN probabilistic model classifier was defined in this work to learn from data collected through a user study, how to infer the appropriate level of autonomy in drone-traffic-control tasks. Participants involved in the study (6 males and 2 females, aged between 24 to 27), were selected from the students of Politecnico di Torino in order to gather data needed for developing a first prototype of the system. A preliminary experiment with 4 participants was conducted to establish a prior subdivision of the number of drones in three different ranges, namely: “Low”, “Medium”, and “High”. In order to do this, participants were invited to monitor from 1 to 6 UAVs characterized by a level of risk linearly proportional to the number of drones. Results obtained showed that a number of drones in “Low”, “Medium” and “High” ranges consists in 1, 2 and from 3 up UAVs respectively.

Afterwards, a brief training phase was performed to instruct participants to act as a real UAVs controller by performing some supervision and monitoring tasks of a growing number of drones. They were invited to monitor and eventually intervene on drones’ behavior by exploiting flight commands showed in the user interface when critical conditions were warned by the UAVs through an alert.

The experiment was organized in six sessions (1 practice and 5 tests) of two trials, one in “Warning” mode and the other in “Suggestion” mode by exploiting the related interface. The above modalities were chosen in a random order so that to limit the effect of learning. Each trial lasted approximately 4 min.

The first test (labeled T1), consisted of a single flying drone whose path was designed for avoiding obstacles on its route. The other two tests T2 and T3 were meant to evaluate the operator’s performance in monitoring two drones flying in a medium bandwidth zone and at risk of colliding, respectively. The fourth test (labeled T4) consisted of three drones, two of which at high risk of colliding and one with a medium battery level. The other test T5 consisted of five drones, three of which at high risk of colliding. Lastly, T6 consisted of six drones, each of which required operator’s interventions to successfully complete the mission. The outcome of each test may be “successfully completed” - if all drones land correctly in the intended positions - or “failed” - if at least one drone crashes. Such tasks have been specifically designed to test the operator’s performance in the possible scenarios he could be involved into in air-traffic management.

During each trial, quantitative data about number of unmanaged drones thus the outcome of each mission as well as information about the “Alert” status of each drone were recorded. At the end of each trial, participants were asked to fill a NASA Task Load Index (TLX) questionnaire [18] for each action performed on the drones. This questionnaire was exploited to evaluate operators’ self-assessed workload on a six-dimensions scale regarding: mental demand, physical demand, temporal demand, performance, effort, and frustration, with a score from 0 to 100. A global score is then calculated by a weighting procedure to combine the six individual scale ratings. At the end of each session (after two trials), participants were also asked to indicate which LOA of the system they preferred in performing the test. For each participant, the execution of the tests and the compilation of the questionnaires took about 2 h.

Fig. 4.
figure 4

Results in terms of (a) percentage of participants able to succeed missions and (b) NASA-TLX average score in the considered missions.

Full size image

Results obtained in terms of number of completed missions as well as percentage of participants able to complete such missions are reported in Fig. 4a. Whereas results concerning average values of the operators’ perceived workload scores are illustrated in Fig. 4b. It can be observed that the percentage of participants able to complete mission T1 is significantly greater compared to the missions T5 and T6. Concerning operators’ self-assessed mental workload, the NASA-TLX average score of mission T6 appeared to be considerably higher than the others. Moving from these findings, operators’ mental workload score in managing 1, 2 or more than 3 UAVs may be labeled as “Low”, “Medium” and “High” workload respectively. These findings corroborate the preliminary results obtained above by confirming the previous subdivision into three ranges according to the number of drones.

Results obtained were then exploited to train the Bayesian Network classifier to learn how to determine the appropriate level of autonomy for the system. Evaluation from the point of view of accuracy was then performed. For this purpose, a cross validation technique was used to test the classification model performance and its ability to predict LOAs on unseen data. According to this validation methodology, data collected were divided into two different groups, namely training set - for training the BN - and validation set - for accuracy validation - as follows: 80% and 20% of the data respectively. Overall data set contains as many rows as the actions carried out by participants on drones. Each row consists of the number of UAVs in the three “Alert” states, the operator’s mental workload level, the outcome of the mission and his/her preferred LOA in that situation. An example of a test result is shown in Table 3. Then the corresponding line for building both training and validation sets is shown in Table 4. The Bayesian Network training phase was performed by exploiting the Netica SoftwareFootnote 8 then the validation methodology was performed by obtaining a classification LOA accuracy equal to \(83.44\%\). Table 5 shows the confusion matrix for each level of autonomy considered in this study.

Table 3. Example of a test result with 3 UAVs.
Full size table
Table 4. Example of a row in the training or validation set.
Full size table
Table 5. Confusion matrix
Full size table

6 Conclusions and Future Work

In this work, an adjustable autonomy system exploiting decision-making capabilities was developed to assist UAV operators by predicting the appropriate LOA relying on operators’ mental workload measurements in drone monitoring scenarios. A Bayesian Network (BN) classifier was exploited as learning probabilistic model and the NASA-TLX questionnaire as subjective workload assessment technique. Obtained results show the proposed model is able to predict the appropriate LOA with an accuracy of 83.44%. Future work will focus on alternative workload assessment techniques, such as physiological measurements, to capture cognitive information in real-time and continually with higher reliability in the measurements.