Keywords

Introduction

At the onset of the 2020s, EU Member States and their non-EU (Third) Country Neighbors have come to share 14.647 km of land borders and 67.571 km of maritime borders (coastline) [1]. The width, landscape biodiversity, and seasonal terrain’s adversities of EU’s borderlands have, before long, come to impact the frontier surveillance duties ascribed to various agencies operating within their premises, that is, National Police Units, Gendarmerie, and Frontex. Faced with such territorial hardships and an ever-increasing number of illegal activities, such as drug and human trafficking, border crossings, and state-backed hybrid threats, Border and Coast Guard Authorities have become increasingly reliant on state-of-the-art equipment (i.e., UAVs, mobile equipment, sensors, machine learning, etc.) to counter the constantly shape-shifting threats [2]. The recent technological advancements in the field of unmanned aerial vehicles (UAVs), often referred to as drones, have emerged as a promising tool in border surveillance given their versatility and mobility, coupled with AI-backed applications and a multiplicity of integrated sensors. In this regard, UAVs can provide EU Border Security Practitioners with significantly improved, timelier, situational awareness, and decision-making capacity. To this end, our work explores an ultra-high-resolution multisensor surveillance payload supporting border surveillance, search and rescue applications, and rough terrain detection(s) using a lighter-than-air (LTA) UAV to complement the efforts of Border Police Agencies during and after the project [3]. By employing a synthetic aperture radar (SAR), shortwave/longwave infrared (SWIR/LWIR) and acoustic cameras for direct target detection, as well as optical and hyperspectral cameras for indirect detection (via vegetation disturbance), this project will use innovative data models (AI) to identify illegal crossing patterns and preferred routes and advanced audio/video analytics and storage in the C2 center to make borderlands safer and more secure. The remainder of this chapter consists of the literature review section and the BorderUAS’ methodology analysis, expanding on the analysis and performance of the payload components. The proofs-of-concept (SAR, acoustic sensor) are also included along with preliminary results. The chapter’s final segment touches upon a series of concluding points and future work until the project’s completion.

Related Work

Border surveillance and control (land, air, and maritime) is conducted by the European or national authorities as depicted in Fig. 32.1 [4].

Fig. 32.1
An illustration of the border management concept along the sea, air, and land. The surveillance is done through marines, drones, helicopters, and radars. The border crossing points from Nation A to Nation B are through sea ports, airports, and check posts.

Border management concept

Full size image

In particular, the case of airborne border surveillance has garnered significant interest from EU agencies. This interest is fueled by remarkable progress witnessed in the drone industry, coupled with advancements in computer vision technology. A comprehensive overview of this tendency is analyzed in [5], where it is stated, among others, that Frontex and the European Maritime Safety Agency began using drones for border surveillance in 2018 after many years of research and pilot projects. The EU-funded program FOLDOUT [6] presented an under-foliage detection method of illegal activity at borders and traces the movement and routes prior to arrival in border areas. To achieve this, a holistic system employing fused advanced ground, aerial, and space-based sensors mounted on Stratobus™ and satellite platforms was deployed in different countries for testing the performance under different weather conditions. In the same context, [7] examines the use of UAVs to support maritime safety and search and rescue operations by providing live video streams and images from the area of operations, within the EFFECTOR project. They developed an embedded system that employs machine learning algorithms, allowing a UAV to autonomously detect objects in the water and keep track of their changing position through time. Another innovative project in the field of sea border surveillance is [8], developing unmanned vehicles (UxV)—aerial, sea surface, and submarine ones, to enhance current maritime border surveillance operations regarding the detection of irregular migrants and narcotics smugglers. In addition to the state-of-the-art, BorderUAS combines for the first time a multirole lighter-than-air (LTA) unmanned aerial vehicle (UAV) with an ultra-high-resolution multisensor surveillance payload, including objects and events detection, providing real-time data stream and storage. BorderUAS also expands on the data harmonization shared among different countries; Ref. [9] developed a knowledge–based model that defines the entities of the land border surveillance domain and establishes potential associations among them, following the preceded work of the EUCISE-OWL project (http://www.eucise2020.eu).

BorderUAS Project Methodology and Payload

The project at hand presents a novelty of having all its payload components mounted on a single platform to allow for interoperability, thus making it stand apart from like-minded ventures mentioned above. BorderUAS payload carries with it two arrays of sensors. Each component is intended to support one (or more) of the border surveillance functions that BorderUAS seeks to empower. Table 32.1 provides a list of sensor functionality and their positioning thereof.

Table 32.1 Overview of BorderUAS’ payload components
Full size table

The footage secured by the payload’s components is then processed by means of an algorithmic design. The feed processing is there to enable end users to determine the degree of a threat each detected event may represent. In a nutshell, the techniques that will be applied to acoustic and visual feedback based on component functionality are laid down as follows:

  • Data acquisition, storage, and streaming

  • Object detection, tracking, characterization, and reidentification

  • Data and metadata fusion for the reduction of false positives or negatives

  • Geo-localization of detections

  • Photogrammetry and multimodal terrain mapping

The payload components, along with the algorithmic design applied in the feed received from them, will both be used in a set of field-trial scenarios designed to test them in real-life conditions. The scenario concepts were the outcome of end user-directed questionnaires [10] and are designated to feature detection/tracking of individuals/objects in open areas, hardly accessible areas or under canopy, as well as indirect (non-real time) detection of individuals and events through observations of the land cover.

BorderUAS Payload-Related Functionalities

The payload components of BorderUAS are briefly described below.

Acoustic Sensor (Proof of Concept)

For the purpose of mapping hardly accessible border areas, especially so during S&R operations, BorderUAS features an acoustic sensor (AS) designed and developed to capture acoustic signal from far distances that can be related to specific events and observations on the ground. Considered are working machinery, passing vehicles, or yelling people. The AS comprises a microphone array and uses a beamforming algorithm to generate a visual presentation of the acoustic field, that is, acoustic mapping of the UAV’s borderland surroundings, in real time.

Besides the acoustic map, which is georeferenced using coordinates from UAV’s GPS, the sensor also allows for real-time augmented listening to specific locations on the ground. The overall concept of the acoustic map acquisition is depicted in Fig. 32.2, where detected ground noise has been geo-localized based on the GPS position of the sensor and the individual results of the beam-forming algorithm.

Fig. 32.2
A 3-D scatterplot and a satellite image. Left, the data is scattered in an elliptical closed-loop form. Right, a location on the road situated between the forest is highlighted.

Concept of the acoustic mapping: the spatial configuration of the AS microphone array adopted to the UAV haul is shown on the left and the resulting acoustic map, marking sound sources, is shown on the right, overlayed on the top of the satellite imagery

Full size image

LWIR/SWIR Novelty

BorderUAS will also feature two IR sensors (LWIR, SWIR) for enhancing border surveillance under nightly conditions. LWIR comprises two cameras, one with fixed and another with a zoom lens, operating at a 7–14 μm wavelength range and will be used for direct detection of individuals/objects via its thermal vision property. SWIR comprises a camera with a fixed lens that operates at a 0.9–1.7 μm wavelength range and uses night vision to detect individuals/objects in ambient lighted terrains and through a glassy surface (i.e., a car’s windshield). The three cameras’ output will be fused (LWIR/SWIR) to allow for freely switching between thermal and night vision whilst tracking the same target.

Hyperspectral Cameras

Hyperspectral (HS) cameras are used for terrain mapping and the case of indirect detection via vegetation disturbance (i.e., stepping on a stuck of tree vines, etc.). Their output is a mosaic type of snapshot that pans out 16 bands of a 4 × 4 grid. Visual as well as near-infrared spectra are captured (see Fig. 32.3). The information captured by the different wavelengths helps in enhancing vegetation imaging by discerning between fresh, dry, or different types of greenery or foliage. An example is the normalized difference vegetation index (NDVI) that is used to quantify vegetation greenness, density, and greenery health. Ultimately, the HS sensor will be used in conjuncture with other payload components in producing multilayered map(s) of a given surveyed area that can be visually assessed by the LEA.

Fig. 32.3
A heatmap and a segmentation illustration. Left, a grid pattern with terrain mapping and vegetation detection highlighting the plantation region. Right, the night vision of various terrains with glassy surfaces.

The result of HS camera’s preliminary testing, presenting NDVI (left) and segmentation (right) analysis of the input data represented as a multilayered map

Full size image

SAR (Proof of Concept)

The land observations through optical imaging have received a lot of attention due to their high spatial resolution and low cost. However, cloudy weather and night observation are critical aspects of optical images. A solution is provided by active remote sensing with synthetic aperture radar (SAR) that works with signal wavelengths that penetrate clouds and smoke.

The most important factor of SAR instruments is the backscattering coefficient that depends on vegetation, humidity, ground terrain, transmit frequency, and the polarization of the frequency. Some of the most common SAR active systems operate in L (1–2 GHz), C (4–8 GHz), and X (8–12 GHz) frequency bands. A collection of SAR ground-based received images is given below. Figure 32.4 shows how the soil and vegetation response varies in all three frequencies. Figure 32.5 shows the contrast between the backscattering coefficients of a water surface and an urban area. The larger the wavelength, that is, lower frequency, the better is the penetration through foliage or ground. To this end, L-band SAR is deployed on the UAV platform that allows for stationary and continuous monitoring of area of interest at the time when a specific event occurs.

Fig. 32.4
An aerial photograph and 3 heat maps. a, a mountain covered by trees and a small deserted land in the center. b, c, and d, range versus cross-range for Ku, C, and L bands plot frustum trends with intensity ranging between negative 10 and negative 40.

Examples of SAR images collected in three different frequencies: (a) scene; (b) Ku-band; (c) C band; (d) L band. The captured area extends in range from 1 to 2 km

Full size image
Fig. 32.5
A photograph and a heatmap of range versus cross range. a, a photo of the coastline with a sunrise and a clear sky. b, plots an intensity between negative 30 and negative 60 with a high intensity in the center.

Ku-band SAR image of a coastal area: (a) scene; (b) amplitude of SAR image. Radar observation from an altitude of 65 m above the scene with a range extension from 2 to 7 km

Full size image

Raw data is processed to generate both full-resolution SAR images and stacks of real aperture radar (RAR) profiles with only range resolution. Both products are georeferenced and are updated about every 5 min. Detection techniques involve target detection via position change made visible in a pair of SAR images through anomaly detection. Additionally, RAR profile stacks generated from a raw dataset allow for target movement detection within the time window of raw data file collection.

Optical High-Resolution RGB Sensor

To best acquire video stream from very large distances, a set of optical sensors that supports a resolution beyond the standards was deployed. Two high-resolution cameras are used, along with a set of interchangeable lenses and one processing unit that serves as the control and processing unit (see configuration in Fig. 32.6). Camera functionalities include, but are not limited to, data storage, preprocessing, live streaming, and perception used for object tracking and characterization of detections.

Fig. 32.6
3 photographs. a, it consists of wide, adaptive, and narrow-range lenses, a computer, an ultra-high and high-resolution camera, and an embedded processing unit. b, location of a car near the buildings. c, 4 photos with detected objects, dominant color, texture characteristics, and trajectory prediction.

HW setup of the high-resolution optical sensor (a). Examples of target detection in the horizon (1 km distance) direction (b) and tracking and characterization of target (c)

Full size image

After the preprocessing of the image data, the data must be stacked in a way for automatic feature extraction, simultaneous detection, and localization of an object. Due to the ultra-high resolution of each input image and the use of a resource-limited embedded system, real-time processing is very laborious. In fact, any modern DNN-based real-time detection algorithm has been designed to handle inputs of much lower resolution. We implemented a three-step optimized scheme for handling ultra-high-resolution images for real-time object detection on our embedded system. We split the input image into K subframes, applying YOLOv5 on each one of the K subframes, and finally merging all the intermediate detections for determining a final detection on the initial input image (for more details, see Antonakakis et al. [11]).

Preliminary Results

BorderUAS platform is an event-driven system of systems. Sensors’ output creates events upon data processing and fusion enhancing them with information from the analytics subsystems. All events and detections are displayed within the Command and Control Centre (C2) for assessment by a LEA operator.

The C2 is a web-based, platform-independent, application designed to enable Border Guard Authorities involved in the future field trials to interact with the UAV platform’s payload and effectively handle events unfolding during scenario execution. The major components featured by the C2 are side-menu, real-time video stream visualization, event list and management, and GIS interface where events, targets, and points of interest are visualized at their global world location (see Fig. 32.7). Additional functionalities supported through particular views are dedicated to sensor and video management, business intelligence, and exporting or reporting on the found incidents.

Fig. 32.7
4 screenshots of the border U A S platform with the restricted area. Laptops are connected to the large television screens. An aerial view of the car highlighted that is passing through the fields.

C2 GIS/event management front-end, introducing live video feed, event notifications, and related map visualization

Full size image

The components constituting the payload of the airborne vessel have already entered into their individual testing phase(s), with some initial results being briefly outlined below.

Acoustic Sensor

Work with the AS so far has involved the use of a beamforming algorithm coupled with array of microphones to improve the signal-to-noise ratio output along with the proper Microphone EMI and wind shielding. Moreover, the use of LMS algorithm can help with noise cancellation coming from the engines of the airborne vessel. Concerning the analysis of received feed will entail K-NN, K-means, & GSOM algorithms to perform noise classification to spot the events of interest.

Hyperspectral Camera

Testing involved the application of principal component analysis (PCA) and support vector machine (SVM) algorithms to perform image segmentation using the HS camera and providing information about the land cover (see Fig. 32.3). Along with the false-color visualizations, these will be used to evaluate the accuracy of image segmentation captured by the HS camera.

MIMO SAR

The added value of the proposed MIMO radar is twofold. First, it can provide SAR images when the LTA UAV is still over the area of interest. Usually, SAR systems need the movement of the platform to generate the synthetic aperture as in the case of space- and airborne SAR systems. In addition, the possibility to generate both an SAR image and stacks of RAR profiles provides a means of detecting anomalies in the scene via a backscattering coefficient every 5 min and tracking the movement of moving targets within the 5-min time window needed to complete the raw data acquisition (see Figs. 32.4 and 32.5).

EOS Sensors (IR and High-Resolution RGB)

Initial test sessions have used the video feeds, along with optimized YOLO4 and YOLOv5 algorithms’ real-time airborne detections of targets. Detection is then followed by object tracking by assigning it a unique ID; this is done using the 2D Kalman filter, along with an association function (OverUnion algorithm), to form a relation between the former’s predictions and the YOLO’s detections. So far, the main objective is to improve the training on detections through the public Stanford dataset and by creating a separate one tailored to the needs of the BUAS scenarios to follow (see Fig. 32.6).

Conclusions

The BorderUAS Project seeks to provide Border Surveillance Authorities with a lighter-than-air UAV platform of 24/7 Operational Autonomy featuring a multiplicity of payload components. The UAV’s payload output is further analyzed and refined by means of algorithmic processes revolving around machine learning and AI support. End users (LEAs) designated to participate in field trials for testing the concept’s operational added value in borderland surveillance will be able to utilize a state-of-the-art Command and Control (C2) System to both have access to payload component feed and help automatize, in an interoperable fashion, the content of alerts, events, and border mission guidelines for in-field units to follow.

Future research expanding on the given EU project’s area of expertise may unfold along the lines of the following topics:

  • Organizing and carrying out field trials to validate the solution at hand

  • Building an ecosystem including industry partners that may wish to employ the payload’s capabilities in the private sector

  • Posthumously testing the bundled payload components on other unmanned or manned airborne/ground vessels to check as to the interoperability, versatility, and viability of the solution at hand

  • Training the models (i.e., for detection, tracking, reidentification, etc.) using additional (public, etc.) datasets for improved sensor/camera output analysis.