Keywords

Introduction

Maritime situational awareness (MSA) is the combination of activities, events, and threats in the maritime environment that could have an impact on marine activities and affect the European Union (EU) territory. European waters are navigated daily by some 12,000 vessels, which share their positions to avoid collisions, generating a huge number of positional messages every minute. It is important that this overabundance of information does not overwhelm the marine operator in charge of decision-making. Thus, the challenge is twofold: (a) on one hand, the large-scale exploitation of heterogeneous data sources, enabling new artificial intelligence (AI)-based services for enhancing MSA; and (b) on the other hand, the seamless integration and exchange of information among maritime authorities valorizing the Common Information Sharing Environment (CISE) network.

The current landscape of AI and big data (BD) is evolving rapidly. Today, AI is heavily driven by data, and machine learning (ML) techniques play a vital role in its success. ML techniques enable machines to learn from data, make predictions, and decisions without explicit programming. These techniques have become instrumental in AI, allowing machines to progressively enhance their performance on specific tasks through experience and exposure to data. Some common ML techniques in AI include supervised and unsupervised learning, deep learning, and natural language processing (NLP).

Supervised learning involves training an algorithm on labeled data to establish a mapping between input features and corresponding output labels. Conversely, unsupervised learning trains algorithms on unlabeled data to discover patterns, relationships, or structures within the data without predefined labels [1]. Deep learning focuses on training deep neural networks (with multiple layers) to recognize complex patterns in data. Deep learning has achieved remarkable success in areas such as image and speech recognition, natural language processing, and generative modeling. NLP aims to enable computers to understand, interpret, and generate human language. These examples highlight a few machine learning techniques, but the field of machine learning is vast and constantly evolving, with new algorithms and techniques being developed to address problems in different domains.

The performance and accuracy of ML models often improve as the available data for training increases. By training on a larger dataset, ML algorithms can capture a more robust and accurate understanding of the problem. The quality and relevance of the data are equally important. It is crucial to ensure that the data used for training is representative of the real-world scenarios and covers a wide range of possible inputs to improve the model’s ability to handle different situations.

With the exponential growth of data, BD technologies have become crucial to handle and process large and complex datasets, commonly known as big data. These technologies are specifically developed to overcome the challenges associated with storing, managing, analyzing, and extracting insights from massive volumes of data that exceed the capabilities of traditional data processing systems. Nowadays, open-source distributed computing frameworks and technologies, such as Apache Hadoop and Apache Spark, enable the processing and analysis of massive datasets across clusters of computers and facilitate real-time data streaming.

Despite significant advancements in AI and big data in various domains, several challenges and gaps remain. Issues such as privacy, safety, transparency, explainability, bias, quality of data, accuracy, robustness, and security have a direct impact on people’s trust in AI technology and their overall interaction experience with AI tools. It is crucial for the ongoing and positive advancement of AI to establish policies, laws, and standardized environments that ensure that AI technologies bring advantages to society and protect the public interest.

To address these concerns, the European Union (EU) is adopting a regulatory framework for AI known as the Artificial Intelligence Act (AIA) [2]. The AIA aims to establish harmonized horizontal rules for artificial intelligence based on a risk-based approach, providing a legal framework for trustworthy AI. It defines a set of objective-based requirements that AI systems should adhere to. Additionally, it establishes transparency rules for AI systems intended to interact with natural persons. According to the AIA objectives, standardization should also play a key role in providing guidelines, requirements, and reference technical solutions to providers to ensure compliance with the regulation.

PROMENADE System Architecture and Deployment

Within the PROMENADE project, various AI-based MSA services have been developed and demonstrated in operational trials; some of them are described in the third section. These services adopt machine learning and deep learning models that have been trained on historic maritime data sets, including Radar, automatic identification system (AIS), images, videos, maritime databases, and OSINT data sources. Large dataset representative of the real-world scenarios has been used in most of the cases to ensure improved performances and accuracy of the models.

High-power computing (HPC) can play a critical role in supporting the development and training of AI models by providing the necessary computing power to process large amounts of data for training and complex algorithms. Accelerating computation, managing big data, optimizing algorithms, and scaling to new architectures are some ways in which HPC can support AI model development and training.

In PROMENADE, Leonardo S.p.A. has made available its own HPC facility DaVinci-1 during the data and computing intensive phases of AI model development and training. Moreover, PROMENADE developed a data lake infrastructure leveraging on open-source big data technologies and acting as the backbone of the system for data ingestion, data storage, data analytics, data, and streaming to integrate diverse data formats processed in the toolkit such as images, videos, JSON, CSV, and CISE data model.

Specifically, PROMENADE has used two different architectures to improve the obtained results and increase the performance of the toolkit: (a) the training architecture deployed as cloud computing infrastructure on top of Leonardo HPC platform DaVinci-1, providing virtualized services for developing the PROMENADE data lake, pipelines, and training of AI services; and (b) the operational architecture deployed at the end users’ premises for trials execution with the already trained services consuming real data and enabling communication with the C2 Legacy Systems and CISE environment.

The architecture of PROMENADE exploits the full potential of the CISE data model to exchange data internally and externally. The architecture is based on microservices that provide a great degree of modularity and flexibility, thus maritime authorities can select which tools they require in their operational domain.

Data is ingested from various sources: for example, (i) video feeds from cameras offering thermal and/or optical streams to detect and classify vessels; (ii) vessel tracks from the VHF Data Exchange System (VDES) or legacy AIS provided by end users or directly from an AIS antenna connection; (iii) satellite imagery obtained to detect and classify dark vessels and correlate with vessel tracks from other sources; (iv) external maritime databases incorporated to enhance the metadata available about a vessel, thereby allowing for risky vessels to be identified early and providing decision support for operations; and (v) additional sensors such as Radar and Radar Direction Finders can be incorporated, thereby providing a fused situational picture and allowing for maximum interpretation by the AI services (see Fig. 30.1).

Fig. 30.1
A network diagram of the PROMENADE architecture. All blocks are connected to the data lake and adaptor layer. The blocks are data fusion tools, video processing, C I S E network, radar, database, A I S, E O S, and analytical tools.

PROMENADE high-level architecture

Full size image

The AI services are categorized according to function and Joint Directors of Laboratories (JDL) categories. A number of services provide input to other services in the toolkit, thereby creating a holistic maritime awareness toolkit. The core component of the architecture is the data lake, which enables the storage of homogeneous and heterogeneous data and publishes this data for PROMENADE services to process and detect situations of interest. The data lake is based on big data technologies enhanced with data transformers and routing mechanisms, providing a transparent area of extract, transform, and load (ETL) processing.

The reference PROMENADE architecture was used as a basis to implement the operational architecture for the Hellenic trial. The deployment took place in Corfu Port Authority and Piraeus, where the national command headquarters of the Hellenic Coast Guard (HCG) is located. The two sites were interconnected via secure VPN ensuring access to the Corfu’s sensors and the legacy C2 mobile (see Fig. 30.2).

Fig. 30.2
A network diagram of the PROMENADE Hellenic trail architecture. The Corfu Island and Piraeus are connected through the internet with secure V P N. Corfu Island has A I S and radar. Piraeus has a small network with fusion tools, a data lake, an adaptor layer, and a database.

PROMENADE Hellenic trial deployment architecture

Full size image

PROMENADE Technological Services

PROMENADE developed a set of innovative AI/BD-based services for im-proved MSA grouped int five main categories: (a) classification services: they provide automated data processing and classification from various sources (e.g., cameras, satellite imagery, and vessel-tracking stations), related to vessel detection, route classification, vessel activity classification, and oil spill detections; (b) pattern detection services: they provide automatic pattern detection applied to multisource data fusion, extraction of patterns of life, behavior analysis, and anomaly detection, etc.; (c) risk assessment services: they provide automated data processing and risk assessment of vessels’ behavior and characteristics through the use of innovative data sources and algorithms; (d) future state prediction services: they allow learning the motion of ships in particular areas of interest from historical data with the goal of predicting their future trajectories and anticipating their future behavior; (e) data infrastructure, which includes the data lake that provides the ingestion, storage, processing, and distribution of data in a BD environment and the data exchange with the CISE network.

Overall, the PROMENADE toolkit includes 22 AI-based and CISE-compliant services. Specifically, the services that have been selected to be demonstrated during the Hellenic trial are described in Table 30.1 per category.

Table 30.1 PROMENADE services tested in the Hellenic trial
Full size table

Hellenic Trial Execution: Results and Lessons Learnt

The Hellenic trial took place in February 2023 in Greece in two locations, Corfu Island and Piraeus. It was attended by around 80 people in HCG headquarters, including policymakers, EU agencies (e.g., DG-HOME, REA, EMSA, FRONTEX, EFCA), and maritime authorities, while more than 30 operational and technical experts participated on the field from Corfu Island. The area of interest was the Ionian Sea around Corfu Island, which is known for its large amount of drug trafficking and in some cases irregular migration. Real operational assets were used by Corfu and Igoumenitsa Coast Guard Authorities to play the role of the facilitators and law enforcement. The scenarios were designed by the HCG to mimic past incidents, thereby validating the use of the PROMENADE toolkit in an operational environment. The main objective of the trial was to make use of AI and BD technologies to detect vessel abnormal behavior and identify a “drop-off” drug delivery at sea.

In order to ensure good sensor coverage of the area, a patrol van donated by the HCG has been entirely renovated and equipped with the following surveillance equipment: (i) an AIS antenna, (ii) an X-Band surface IP radar system with 48 nautical miles (NM) range, (iii) a satellite compass, and (iv) a long-range IR—Thermal Internet Protocol (IP) Pan-Tilt-Zoom (PTZ) camera. Furthermore, a display-and-control unit has been installed on the co-driver’s side to allow the co-driver to control the camera, focus on the regions/objects of interest, etc. (see Fig. 30.3). It is noteworthy that the driver is not distracted by the light and brightness of the screen and the actions of the co-driver as the installation has been carried out in a way that prevents any such issues. These capabilities are leveraged using the ENGAGE platform, which enhances the end users’ experience by offering them the opportunity to access and process the data of interest via mobile devices, such as tablets and laptops apart from the multifunctional plotter. The ENGAGE Border Management Edition (BME) developed by Satways Ltd. is a fully fledged C2 operated by the Hellenic Coast Guard for various R&D activities ensuring the latest cutting-edge technologies are tried and tested by their personnel.

Fig. 30.3
2 photographs of the surveillance equipment. Left. A camera, a sensor, and antennas are installed on the H C G patrol van. Right. A monitoring screen to observe activities.

Photographs of the surveillance equipment installed in the HCG patrol van

Full size image

In addition, there is equipment installed in the rear cabin of the van (see Fig. 30.4). An alternating current (AC) power as well as a 12 V, 120 Ah power system battery (PSB) are also installed in the rear cabin. The additional battery ensures the smooth functioning of the patrol van and guarantees it works independently from a power autonomy standpoint as the battery is charged when the engine of the van is on. Of course, the system is designed in such a way that the engine does not need to be on as long as the battery is adequately charged. For example, the system can work for almost 15 hours with the engine of the van constantly off.

Fig. 30.4
2 photographs. Left. Telecommunication equipment with sensors, some electronic boxes. These are connected to the power system. Right. A power system with a battery and U P S in a cabin.

Photographs of the telecommunication equipment and the power system battery installed in the rear cabin of the HCG patrol van

Full size image

The appropriate equipment has been installed on the van and customized, leading to its transformation into a self-contained, highly autonomous and portable command-and-control (C2) system. This system aims at equipping the operational staff with all the capabilities, which are considered necessary for carrying out their operational duties, especially in cases where the required infrastructure for the support and deployment of a C2 system is unavailable (e.g., in remote and/or isolated areas). Therefore, the van constitutes a modular, easy-to-deploy, but secure design, capable of satisfying the end users’ needs as a portable C2 center integrated with the HCG legacy systems. This goal is achieved by offering state-of-the-art patrolling, surveillance, detection, risk management, and future state prediction capabilities, enhanced by advanced AI as well as data fusion techniques, incorporated into the PROMENADE toolkit.

Ultimately, the operational goals through the execution of the Hellenic trial were related either to apprehend activities with drugs loaded and unloaded in remote areas with vessels that are fast enough to avoid detection or capture, or dropped off drugs that can be transported by a range of commercial vessels such as cargo ships, fishing vessels, tankers, and tugboats, so as to be collected by the drug traffickers. The overall outcome of the AI services tested with real vessels during the Hellenic trial was very positive while the main lesson learnt was related to the dynamic adjustment of the system alert parameters to depict the 100% operational challenges that the end users have to face in the field. End users’ feedback was collected through a questionnaire developed using the EU survey. Overall, the operators noted that the use of AI and big data will be a significant advantage in upgrading deprecated systems and achieving early detection of targets through valid patterns of abnormal behavior.

Conclusion

In the domain of maritime surveillance, PROMENADE project has made significant advancements in the field of MSA, enhancing border and external security capabilities by developing and integrating in a unique open and CISE-compliant service-based toolkit a set of innovative AI/BD-based services.

By leveraging advanced technologies, integrating diverse data sources, and conducting extensive trials, including the Hellenic one, PROMENADE has paved the way for enhanced maritime security, decision-making, and information exchange among maritime surveillance authorities. The project’s achievements in providing valuable insights, predicting vessel behavior, and enabling efficient data processing will contribute to the overall goal of ensuring the safety and security of European waters and territories.