Keywords

Introduction

The EURMARS project enhances EU efforts in addressing increasingly complex security risks and threats regarding border management in the maritime domain by designing and implementing a multiauthority border surveillance platform that integrates AI, risk assessment, and visualisation innovations supported by advanced sensing technologies, such as high-altitude platform systems, satellite imagery, unmanned vehicles (UAVs), and ground-based sensors. The main end users to benefit from the project results are border authorities (BA) and agencies at the national and EU levels, such as coast guard, border guard, customs, police, fisheries control, environmental protection, and maritime safety entities. The overall concept is depicted in Fig. 31.1.

Fig. 31.1
A conceptual framework for E U R M A R S. It has 3 layers. The top layer has satellite-based data, the middle layer has maritime structure, and the bottom layer has E U R M A R S mission-deployable assets.

The EURMARS concept

Full size image

In this chapter, we describe the innovative methodological approach and the transformative integration of AI, risk assessment, visualisation novelties, and advanced sensing technologies, including high-altitude platform systems, satellite imagery, unmanned vehicles, and ground-based sensors as a blueprint for the development of next-generation 24/7 border surveillance platforms. Key innovation enablers are founded on solid and standardised best practices for user requirements extraction and extend to the development of enhanced data collection and analysis mechanisms (data fusion), improved risk assessment, real-time visualisation and monitoring, with integrated decision support system offering tailored insights and improved efficiency in situation awareness and operational capacity for a wide range of maritime security risks and threats in remote environments. The proposed blueprint additionally includes the operational validation in real-life scenarios, an AI Foresight Report and Blueprint containing PIA, EIA, and SIA assessments, and a best-effort attempt to create open-source domain-specific benchmark datasets and contribution to relevant international standards. By committing to implement and validate the presented approach, EURMARS aims to achieve significant advancements in data-driven decision-making, standardisation initiatives, and relevant research, with the potential to transform the border management domain and significantly improve border security.

The viability and effectiveness of the EURMARS concept will be evaluated in the following pilot use cases: (1) PUC1—Maritime Border Control: Detection of Trafficking and Other Illegal Activities; (2a) PUC2a—Search & Rescue; (2b) PUC2b—Maritime Structures and Oil Spills Surveillance and Monitoring; and (3) PUC3—Land Border Control; Illegal Crossing Outside of Business Contingency Plans.

Motivation of PUC1 is to improve the detection of abnormal behaviour of vessels involved in illegal activities, with a focus on small boats near shipping routes and shorelines. Existing controls are insufficient for detecting small boats approaching the shoreline, leading to difficulty in identifying illegal activities amidst legitimate traffic and irregular coastlines. A demonstration will be conducted to address illegal activities, including the transfer of trafficked people from larger to smaller vessels. The system’s provision of timely alerts regarding high-risk vessels to Border Force Operational Units allowing appropriate actions to be taken will be evaluated.

In PUC2a, EURMARS enhances search and rescue (SAR) operations, aligning with EU definitions of rendering assistance to distressed vessels and individuals at sea. EURMARS utilises satellite images, video, SAR data, and AIS information to detect the relevant scenarios, enhancing the effectiveness of search and rescue forces. Various satellite missions (including Copernicus, NEMO-HD μ-Satellite) enable a long endurance and wider surveillance capability to situational awareness.

The objective of PUC2b is to consistently monitor offshore structures, including gas and oil platforms, detecting emergency situations such as fires, structural failures, terrorism, armed attacks, and oil spills to prevent human loss and mitigate environmental impact. The integration of high-altitude sensors offers a groundbreaking capability for regular or real-time monitoring of structures far from land, overcoming the limitations of current systems that rely on expensive and timely helicopter deployment. The system’s rapid response capability allows SAR operations to mobilise forces, including boats, helicopters, and UAVs, for timely assistance to offshore platforms in emergencies, such as evacuations or medical support. The platform’s ability to detect oil spills promptly contributes to mitigating their effects on natural habitats, preventing potential environmental crises.

The challenge in PUC3 is countering illegal migration and human trafficking at state borders, specifically involving individuals arriving by boat and then moving inland to predetermined locations for illegal border crossing. EURMARS enables border guards to follow illegal activities in a joint multiauthority/country cross-border deployment. As maritime and land border authorities currently lack the necessary resources to disrupt criminal networks, the deployment of EURMARS allows the realisation of seamlessly tracking handovers between sea/land and land/sea, a capability not possible with existing systems.

Methodology for Eliciting Requirements for Border Surveillance

The requirements engineering methodology has been an integral part of the project’s innovation by combining leading requirements engineering practises and standards [1, 2], with insights and lessons learned gained from similar prior activities that align with the project’s objectives, the available time frame (6 months), and the iterative development cycle adopted in the project including the prototype development and living labs establishment.

The adopted methodology included four phases: requirements elicitation, requirements analysis, requirements specification, and requirements validation. The requirements elicitation phase involved collecting requirements from a variety of stakeholders, including end users, domain experts, and technical experts. The requirements analysis phase included interpreting the collected requirements and identifying any inconsistencies or ambiguities. The requirements specification phase focused on creating a formal document that describes the requirements in detail. The requirements validation phase involved verifying that the requirements are complete, consistent, and feasible.

The methodology (see Fig. 31.2) was implemented using a variety of techniques, including surveys, interviews, workshops, document analysis, and prototyping. The requirements were iteratively refined and prioritised throughout the process of collecting, analysing, and converting the inputs from the stakeholders into exact requirements specifications, ensuring that they comprise the common needs of end users. High emphasis was placed on co-creative practices that intensified towards the end of the overall requirements development process. Especially for requirements prioritisation, partners were asked to provide feedback individually, and the overall material was discussed and processed together in workshops to achieve a shared view on the importance of each requirement. The process implemented to develop the requirements is illustrated in Fig. 31.2 and resulted in the specification of over 140 individual requirements structured into to nine high-level categories.

Fig. 31.2
A flow diagram of the methodology used for E U R M A R S requirements development. It begins with a literature review and the drafting of an end-user survey, and ends with the processing of iteration round feedback and the finalization of the requirements list through various steps.

Research process for EURMARS requirements development

Full size image

Modern Sensors for Border Surveillance

In its core, the EURMARS platform incorporates a combination of novel AI-powered coastal ground sensor platforms, low- and high-altitude sensing systems, and satellite-based systems to feed the advanced fusion and reasoning functionality. In complement to classical optical sensors that work on different wavelengths and at different altitudes, existing relevant data sources (e.g. vessel transponders) are additionally being analysed and integrated to improve situational awareness.

Coastal Ground and Low-Altitude Sensing Systems

EURMARS is developing a mast-mounted edge-based sensor platform enabling automated vessel/boat, person, and ground vehicle detection and tracking for all different weather conditions with a focus on challenging foggy maritime situations. This is achieved by filling the classical spectral gap between visual (RGB) cameras and thermal sensors with a new sensor platform integrating SWIR and UV sensors. An AI-based algorithm is used for the detection of small vessels up to 1 km from the coast and people and vehicles near shore (YOLOX for vessel detection and other YOLO [3] derivatives for automated detection). Detections are fed to a tracking algorithm to generate short-term tracklets for vessels and ground objects. The same sensor platform will be used for classifying the vessels—a set of residual neural network models have been trained on various representative maritime vessel/ground vehicle training sets, with data augmentation where appropriate to increase the robustness of the models.

To enhance the platform’s detection capabilities of targeted objects (including small vessels) in the distance of a few kilometres, EURMARS employs low-altitude sensing systems in the form of UAVs (fixed-wing, multirotor, and helicopter types) equipped with surveillance sensors such as electro-optic, thermal, and hyper-spectral cameras. UAVs are activated and deployed when unconventional occurrences are identified by other sensors like camera subsystems and satellite-based systems. In its operational role, the UAV platform, during its patrol flights, corroborates and validates events (such as vessels, individuals, and oil spills) monitoring their progression (as long as the UAV flies within the targeted area) while supporting the streaming of live video feed from the designated area. The available infrastructure integrates a robust AI-enabled computing system that allows real-time identification of events and objects of interest like ships, small watercrafts, and individuals within the maritime environment. The generated data include types and dimensions of the identified objects, precise location, trajectories, and count of individuals present in the water. The platform transforms this data in identified and categorised incidents, along with their associated metadata encompassing geographical coordinates, images, classifications, and other details.

High-Altitude Sensing Systems and Satellite-Based Sensing

The sensing assets providing the highest value in the EURMARS innovation proposition are the satellite-based systems which significantly contribute to the expansion of the platform’s capabilities. Main functionalities include vessel and oil spill detection through the use of both synthetic aperture radar (SAR) and optical data from the Copernicus hub (Sentinel 1 and 2) [4], Landsat 8 [5], Landsat 9 [6], the ICEYE commercial satellite [7], and NEMO-HD (cubesat) [8]. AI/ML models are used in combination with the YOLO [3] algorithm to provide event detection in multiple output formats. Additionally, EURMARS will develop earthLIVE, a particular simulator for a telescope equipped with COTS detectors, baffles, pointing mechanisms, and video telemetry transmitting equipment. The simulator will take as input videos from NEMO-HD of up to 3 min in duration, captured in low-resolution (40 m) and high-resolution (2.8 m) mode, and generate vessel detection events.

Relevant Data Sources Intelligence

To complement the spectrum of sensing systems from the perspective of external data acquisition, EURMARS will harness open-access data sources to enhance actionable intelligence for its purposes. The types of open data that can be processed include (i) VDES/AIS data, (ii) weather data, and (iii) CISE data. Regarding VDES/AIS data, a common data hub infrastructure exploits data acquisition from three types of sources: EURMARS’ own receivers installed at the pilot sites, Cypriot and Bulgarian BAs’ infrastructure and prominent commercial services (marinetraffic.com, vesselfinder.com, etc.). Present and forecasted weather data are obtained through the Copernicus Climate Change Service (C3S) [9] which provides information about the past, present, and future climate conditions in Europe. EURMARS also plans to develop CISE adaptors to be able to connect to the CISE network [10], an EU initiative promoting interoperable data exchange between concerned authorities for missions at sea. Envisioned information retrieval will regard information on vessel position, incidents, and alerts on vessel trajectories.

Advanced Fusion, Reasoning, and Risk Assessment for Border Surveillance

The data acquired from the sensing systems and the external data sources (candidate vessel detection positions, tracks, movement directions, object sizes and speeds, vessel/vehicle classification, velocity signatures, etc.) is fed into the multimodal data fusion platform (MDFP) for processing of the information into a single authoritative source for further analysis. A diagram of the MDFP and its connectivity with other modules within the EURMARS system is provided in Fig. 31.3.

Fig. 31.3
A flow chart of the multimodal data fusion platform. The costal ground and low altitude sensing systems, high altitude sensing systems, earth live platform, satellite based platform, and common data hub lead to the primary fusion module, which leads to the feedback loop and tracking module.

The multimodal data fusion platform and its interfaces to other components

Full size image

The MDFP comprises three separate subcomponents: primary fusion module, fusion tracking module, and feedback loop.

  • The primary fusion module performs fusion by using a Bayesian inference system coupled with computer vision techniques [11]. It uses probabilistic occupancy maps, similar to occupancy grid maps used in robotics, to aggregate information from the sensors. Subsequently, areas of high confidence are detected to generate fused detection events.

  • The fusion tracking module parses the fused detection events to provide tracking of targets over time. The module provides multiple levels of tracking algorithms with custom rules for target recovery specific to the current scenario, for example, associating multiple tracks together if they contain the same identification via AIS. The tracked target information is then forwarded for behaviour analysis and risk assessment.

  • The feedback loop takes the events from the previous modules and determines based on confidence scores whether additional information is required. In cases of low confidence, the relevant sensing systems are prompted to reposition or reorient to capture higher quality information; for example, reorient a pan-tilt-zoom camera towards a specific geo-location.

The MDFP can operate in two modes depending on the scenario: a live pipeline that directly processes events from available sources, and an offline mode that can use recorded information. This offline mode enables the inclusion of detected events from sensors with a large delay—such as high-quality satellite-based sensors that have a multi-hour lead time.

The final step in data processing entails the risk management, an essential procedure for identification assessment and mitigation of the potential threats that may affect the security of borders and the safety, and well-being of individuals. Existing risk assessment involving vessels for situational awareness is typically probabilistic or fuzzy-based and lacks qualitative risk assessment. EURMARS’ risk assessment framework is based on FRONTEX’s Common Integrated Risk Analysis Model (CIRAM) [12] that provides a standardised methodology for conducting risk analysis at the EU level and supports the operational planning and decision-making of BAs. EURMARS’s risk assessment framework is a flexible data-driven vessel risk profiling tool adaptable to different scenarios and contexts. It can analyse the current situation and detect threats based on the described data fusion and assign ranks to the threats based on their potential impact for the present time and probable states in which they may evolve. Core functionality includes the automatic recognition of threats, the analysis of potential impact of threats focusing on cross-border threats and threats that may have a larger impact or an important humanitarian impact, requiring coordination and collaboration of multiple authorities.

Ethical Aspects

The ethical perspective is of utmost importance in EURMARS as the project is committed to safeguarding the ethical dimension of all procedures and results. These dimensions comprise the review of all relevant legislation, EC recommendations and best practices, coverage of all the related ethical requirements, identification and recruitment of research participants, ethical approvals for research with humans, personal data processing, and Human/Fundamental Rights Impact Assessment (HRIA/FRIA) related to AI. Procedures and criteria have been identified and utilised to identify activities requiring an ethics approval prior to commencement, to manage delivery of activities’ ethics approval and devise project-tailored informed consent procedures and informed consent forms. In addition, a detailed review and analysis has been performed on all procedures entailing information on processing of personal and sensitive data. The data minimisation principle has been enforced by the detailed design and description of technical organisational and security measures implemented by partners to safeguard the data and documentation of anonymisation and pseudonymisation techniques. Finally, due to the development of AI technologies, based on the HRESIA methodology for AI [13] and the EC guidance on trustworthy AI [14], a framework was designed for HRIA/FRIA and project-specific AI-related Risk Assessment (AIRA) covering the development, deployment, and post-deployment phases of EURMARS, including detailed information on how respect for fundamental human rights and freedoms will be ensured. The framework ensures that the artificial intelligence (AI) components will be integrated through an ‘ethics by design’ (EbD) [15] approach, conforming to all current applicable regulations and including recommendations regarding measures for risk prevention, minimisation, as well as mitigation. The alignment with EU’s AI-Act [16, 17] is evaluated in an AI ACT Foresight Compliance, Social and Ethical Impact Assessment report.

Conclusions

The EURMARS platform emphasises giving BAs the tools to track, identify, and classify vessels and, on occasion, people and ground vehicles in real time over a large geographical area in a robust and accurate way by engaging a set of sensors that range from low to high altitude and space technologies, leveraging private (earthLIVE, Stratobus), as well as national initiatives (Nemo-HD) for maritime surveillance. EURMARS covers the interface between land and sea operations and in doing so enhances the collaboration of the relevant actors via the technical infrastructure that integrates the information and makes it available to them. Besides the technical and situation awareness innovations, the project boasts specific activities to deliver on the ethics capacity, through impact assessments, recommendations, and training materials and contributing to ensuring the protection of human rights with respect to emerging technologies across and beyond Europe. In the long term, EURMARS envisions to contribute and support Actions of the EU Maritime Security Action Plan [18] by ‘investigating synergies with the civilian sector, also harmonising the system requirements’ and by contributing to the development of a cross-sectoral agenda for maritime security research and the dual-use of technologies contributing to maritime security’.