Keywords

Introduction

Given the increasing interconnection among infrastructures, networks, and operators delivering essential services across the internal market, it is necessary to fundamentally reinforce the resilience and operation of the critical entities that manage and operate them [1]. One of the main infrastructure domains that provide essential services is that of transport, which can result in significant and lasting degradation of ecosystems and habitats. Considering that Europe has the highest transport infrastructure density in the world, there is an urgent need to address this rapidly increasing challenge. Road infrastructure tops the list of all transport infrastructures and public assets. Roads are crucial for economic development and growth, providing access to education, health, and employment. The maintenance, repair, and upgrade of roads is one of the most important parts of their high-level service provision and business continuity.

At a time of zero tolerance (zero accidents, zero operating restrictions, etc.), it is increasingly necessary to control risks and to improve the knowledge of the condition of structures to organize preventive and/or predictive maintenance that minimizes risks at an acceptable cost. In specific, instrumentation and risk analysis approaches allow to better understand the behavior of structures, to know their condition, and thus to provide reliable input data for robust risk analysis.

To meet the above needs, the HERON project will develop an integrated automated system to perform the maintenance and upgrading roadworks, such as sealing cracks, patching potholes, asphalt rejuvenation, autonomous replacement of CUD elements, and painting markings, but also supporting the pre/post-intervention phase including visual inspections and dispensing and removing traffic cones in an automated and controlled manner. In turn, this will reduce accidents, lower maintenance costs, and increase road network capacity and efficiency.

More specifically, for coordinating maintenance works, the project will design an autonomous ground robotic vehicle supported by autonomous drones. Sensors and scanners for 3D mapping will be used in addition to artificial intelligence toolkits to help coordinate road maintenance and upgrade workflows. All the above components combined with several other technologies will be integrated into a decision support system (DSS), providing decision-makers, operators, and field crew with all the information required to organize their operational procedures and execute successful road inspection and decision-making activities.

The needs and expectations of the road infrastructure operators are presented in the following sections, based on which the overall architecture of the HERON solution has been designed. More details are provided for those components responsible for the decision-making process and the operational picture during the missions.

End User Needs and Expectations—The Greek Pilot Case

The main end users of HERON are ACCIONAFootnote 1 and OLYMPIA ODOSFootnote 2 the latter of which constitutes a Motorway Concession Project of particular strategic importance on the national and regional level for the development of the Peloponnese and Western Greece, as it connects Athens with North Peloponnese, Western Greece, and the Port of Patras (Fig. 15.1).

Fig. 15.1
A map of Olympia Odos. It represents the motorway with major cities. It starts from Elefsina to Pyrgos bia Thive Elesfsina, Megara, Panorama, Korinthos, Tripoli, Kiato, Derveni, Akrata, Aigio East, Rio, Patra, Varda, and Amaliada.

Map of Olympia Odos Motorway

Full size image

OLYMPIA ODOS provides operation and maintenance services, such as toll collection, traffic management & safety and routine maintenance. Their main needs include [2]:

  • to prevent its early wear and restore any damage, wear, or malfunction may be presented in an effective and efficient way;

  • to operate at a high level of service and to keep a smooth and continuous traffic flow under normal operation conditions, maintain these flow conditions, and minimize delays;

  • to zero the incidents with implication of its personnel;

  • to save natural resources, prevent pollution and reduce its negative environmental impact, and protect third parties’ assets, in the areas of the company’s operation.

Based on the end user needs, the following key specifications have been identified (Table 15.1).

Table 15.1 HERON specifications
Full size table

Based on the above needs and specifications, the following architecture and technical components have been identified, defined, developed, and will be evaluated through three piloting activities.

HERON Solution

To coordinate maintenance works, HERON project will design an integrated automated system which consists of the following components as further depicted in Fig. 15.2 [3].

  1. (a)

    UGV actuators perform various actions related to maintenance missions (cone placement/removal, painting road markings, pothole fill, crack repair, etc.)

  2. (b)

    Sensors, embedded as payloads of UGV and UAV platforms, capturing visual information with regard to the road infrastructure

  3. (c)

    Robotic system, performing all actions related to planning, navigating, and controlling the robotic system and its actuators

  4. (d)

    Secure data communication, enabling secure and seamless connectivity of the various HERON subsystems

  5. (e)

    Middleware & data fusion, enabling information flow and interaction among the various HERON components

  6. (f)

    Sensing Interface & AI, including data acquisition from the various UGV/UAV inspection sensors, data analysis, and information extraction, to perform object detection and semantic segmentation of the road infrastructure

  7. (g)

    Incidence Management (IMS) & Decision Support System (DSS), enhanced with Augmented Reality (AR), providing decision-makers, operators, and field crew with all the information required to organize their operational procedures and execute successful road inspection and decision-making activities.

Fig. 15.2
A tree map of the HERON architecture. The data is gathered from city resources, externally connected devices, and other devices. It presents 3 module layers preprocessing with resources, main processing with management, and module processing with HERON major services.

HERON high-level architecture

Full size image

The following sections focus on those components that aim to enhance their situational awareness and execute successful road maintenance missions and decision-making activities.

HERON Middleware

HERON middleware will incorporate information from various HERON components and sensors and will interact with the application layer (Layer 3) in an appropriate format while ensuring high availability and scalability. It consists of two layers:

  1. I.

    Layer 1—Preprocessing: At this layer, the necessary data preprocessing actions take place. Middleware will be accepting information from various sources that will be consolidated in common smart data models keeping a unified data scheme. HERON will utilize available FIWARE data models using JSON Schemata for the so-called key-value representation of context data.

  2. II.

    Layer 2—Main Processing: At this layer, the various data coming from Layer 1 are stored, processed, and served to Layer 3, which constitutes the application level. Layer 2 also includes the virtualization of data into objects as well as their normalization and storage. Moreover, the processing is tightly coupled with resource management to produce further events and handle the data fusion process. Through Event Management, filtering and contextual information modules will handle all the events from the data sources and will categorize retrieved events and information to proper categories for better understanding and processing by the HERON platform. The events and data will be also transformed, stored, and analyzed to produce additional events and data aggregations that will enrich the system. The fused information will be available through a corresponding developed API to the DSS system and the other modules that require additional data. Finally, raw data will be available to all high-level modules and applications for further and more application-driven processing.

Following Fig. 15.3, the Middleware will stand at the core of the HERON system and its ultimate role is to ensure data integrity by accepting and storing sensor data from trusted sources and data security by allowing access only to authorized requests. It includes a tailored policy-based management framework along with suitable enforcement mechanisms dealing with data encryption, access control, privacy, and anonymity. Furthermore, this block will include intrusion detection and prevention mechanisms, such as tools dealing with protocol analysis, detection of anomalous behavior, security events, intrusion detection, vulnerability assessment, and honeypots. It also includes knowledge repositories and distributed threat registries.

Fig. 15.3
A tree map of the architecture of HERON. It represents data gathering from different sources and flows from top to bottom. The data layers are followed by U G V sensors, a sensing interface, and A I, secure data communication between middleware and robotic systems, and U G V actuators.

Overall middleware and data fusion architecture of HERON [3]

Full size image

The appropriate interfaces/protocols (e.g., sftp, kafka broker, HTTP/HTTPS, MQTT/AMQP) for communication with the different data sources and user services will be created. The module will handle seamlessly aspects such as time synchronization, scheduling, selection of communication paths, fault tolerance, and traffic shaping.

The main input for the middleware will be the information coming from the Sensing Interface and AI component where the information from the sensors from UGV and UAV is acquired and analyzed, along with data provided directly by the road operator such as inventory data, traffic data, and meteorological data. Especially in OLYMPIA ODOS pilot, various types of data provided by the RI operator, such as Traffic Data, Meteorological Data, data coming from Dynamic Message Signs system and maintenance work data will be transferred to the middleware via an appropriate developed sftp protocol. All this data will be further processed creating the output to IMS/DSS.

Finally, the output produced by the HERON middleware will constitute, using APIs that follow a JSON standard, the input to the IMS and the DSS discussed in the next section, as well as the robotic system interface (e.g., the UAV/UGV mission).

Incident Management and Common Operational Picture

The HERON system aims at providing extended situation awareness to key stakeholders during road maintenance and road inspection operations. Situation awareness is the perception of environmental elements and events concerning time or space, the comprehension of their meaning, and the projection of their future status. HERON approach aims at utilizing three main components to grant users a real-time information stream that will enhance their situation awareness: Common Operational Picture (COP) module, AR app, and IMS&DSS App. The aim is to provide the decision-makers, operators, and field crew with all the information required to organize their operational procedures and execute successful road inspection and decision-making activities.

More specifically, the IMS will be based on the light client of ENGAGE IMS [4] extended to interconnect with the HERON Middleware and to support the specifications and business logic of the HERON use cases. The IMS/DSS system is based on a containerized architecture that makes it possible to package software and its dependencies in an isolated unit. The IMS light client takes advantage of the speed and portability of web-based applications to display all the necessary information to the user with all information processing handled on the server-side of the system. The connections are handled with the reverse proxy server as the most efficient way to link to a web server from a remote location. This approach protects the HERON DSS/IMS server from direct connections from an outside source, performs load balancing, keeps track of requests made, and provides a level of anonymity for the server for cybersecurity purposes. The storage of the data will be handled with a database server that is capable to receive heterogeneous data from multiple sources, processing it to obtain normalized and aggregated data and storing it in a distributed resilient file system to be ready for consumption by IMS/DSS system. With these modules, the IMS will generate and share a COP among RI personnel and relevant road authorities permitting the collaborative response of all involved relevant local and regional partners when needed. For maintaining effective communication, facilitating the process, and ensuring unity of effort, the IMS will utilize protocols for multi-level and multi-actors’ interaction.

The COP will act as the central and virtual representation of the HERON Robotic platform controller, providing the Robot operators and decision-makers of the RI companies with all the information required to successfully organize their tasks. The various COP elements will be decomposed into information layers and categories to allow for a flexible system that permits the “need-to-know” principle as different users and roles are envisaged to interact with the HERON tools.

The AR system will provide real-time visual information on the surrounding environment of the robot operators. The AR app software will visualize the number of existing defects (automated detection of pavement defects and classification of severity) and will use overlays of 3D models to display possible hidden structural elements which can affect the maintenance process or additional damages. Display of functional elements will be available through appropriate commands as well.

The above components (see Fig. 15.4) will be interconnected with HERON Middleware to support the specifications and business logic of the use cases. Furthermore, the middleware will process data related to UAVs and UGVs (robots) and sensor telemetry data, as well as data from other digital assets in the maintenance field. This data will be processed, analyzed, and used, combined with 3D data of the scene on the field, to provide the necessary information, on the maintenance and inspection, through user interfaces with 3D visualization and GIS capabilities. In addition, it will provide the high-level mission and tasks (through Middleware) to the robot operators for further actions to be accomplished.

Fig. 15.4
A block diagram of the HERON middleware components. It presents a C O P block connected to I M S, D S S, and A R blocks. The I M S block presents a strategic decision-maker and field manager. The A R block presents a robot operator.

IMS, COP, AR user, and roles

Full size image

Conclusions

In this paper, the authors present the concept and added value of an integrated and autonomous system for the inspection and maintenance of road infrastructures. In specific, the proposed system called HERON is expected to:

  • improve the cost of maintenance activities, by reducing mainly the required human resources;

  • reduce the time period of road/lane closures and the relevant road users’ annoyance;

  • minimize personnel’s exposure to risks due to both maintenance activities and adjacent traffic;

  • minimize environmental pollution and ensure sustainability.

Respective KPIs will be used as the basis for evaluation procedures both at the Spanish and at the Greek pilot case which are planned during the next (and last) phase of the project. Meanwhile, technical developments are continuously implemented, fulfilling the user needs, as described above, but also integrating the legacy system of the operators to provide a holistic solution through a unique interface. Such equipment and infrastructure include fleet management systems, Variable Message Systems (VMS), Traffic Management Center, CCTV, meteorological sensors, Tunnel monitoring systems, etc.