Hastily formed knowledge networks and distributed situation awareness for collaborative robotics

In the context of collaborative robotics, distributed situation awareness is essential for supporting collective intelligence in teams of robots and human agents where it can be used for both individual and collective decision support. This is particularly important in applications pertaining to emergency rescue and crisis management. During operational missions, data and knowledge are gathered incrementally and in different ways by heterogeneous robots and humans. We describe this as the creation of Hastily Formed Knowledge Networks (HFKNs). The focus of this paper is the specification and prototyping of a general distributed system architecture that supports the creation of HFKNs by teams of robots and humans. The information collected ranges from low-level sensor data to high-level semantic knowledge, the latter represented in part as RDF Graphs. The framework includes a synchronization protocol and associated algorithms that allow for the automatic distribution and sharing of data and knowledge between agents. This is done through the distributed synchronization of RDF Graphs shared between agents. High-level semantic queries specified in SPARQL can be used by robots and humans alike to acquire both knowledge and data content from team members. The system is empirically validated and complexity results of the proposed algorithms are provided. Additionally, a field robotics case study is described, where a 3D mapping mission has been executed using several UAVs in a collaborative emergency rescue scenario while using the full HFKN Framework.


Introduction
The importance of effective communication and efficient data/knowledge transfer is essential for the coordination of life-saving activities in regions affected by natural or man-made disasters. The organizations and groups involved range from disaster relief, governmental and non-governmental organizations at the macro level, to actual teams of emergency rescue responders on the ground at the micro-level. Emergency rescue teams have recently been supported by heterogeneous robotic assistants. There are an increasing number of natural disasters that include wild fires, hurricanes, earthquakes and floods that require state-ofthe-art emergency response to minimize loss of life and property damage.
In a seminal article, Denning [32], pointed out the importance of establishing Hastily Formed Networks (HFNs) in the broader sense as "the ability to form multi-organizational networks rapidly" and as being crucial to disaster relief. Here, Denning's focus was on effective human communication rather than efficient data/knowledge transfer, and the use of autonomous systems in emergency rescue operations was not yet prevalent. Additionally, the term conversation space was introduced for the medium in which such communication takes place.
Experience has shown that first response is dependent on the quality and nature of this conversation space. This space is intended to provide a medium for acquiring situation awareness and the original concept was very much focused on setting up the physical layer for communication. Proposed components of the conversational space were the physical systems, the players involved and the interaction practices. The latter include situation awareness, acquiring and sharing information, planning, making decisions, coordination, and command and control required by the players and teams involved.
Using Denning's metaphor of Hastily Formed Networks for enhanced communication and conversation spaces among human agents in emergency rescue operations as a starting point, we extend the idea in two ways using the term Hastily Formed Knowledge Networks (HFKNs) as a guiding metaphor for this research.
• Firstly, rather than focusing solely on teams of human agents, we focus on teams of human and heterogeneous autonomous robotic agents interacting in various ways among themselves and with humans to make rescue operations more efficient and to achieve collaborative goals.
• Secondly, rather than just focusing on communicative aspects and conversational spaces for crisis communication, we instead focus on both communicative aspects and data/knowledge aspects in data/knowledge interaction spaces among collaborative teams consisting of both human and robotic agents.
The goal is to automate the goal-driven creation and exchange of local and global situation awareness among team members (both human and robotic) in addition to improving the basis for informed decision making by providing timely data and knowledge rich contexts for doing this. These data and knowledge contexts will be both individually and collaboratively constructed on-the-fly during the unfolding of emergency rescue missions relative to the needs and requests of human and robotic team members.
The work presented in this paper is part of a larger infrastructural multi-agent based framework being developed with the goal of leveraging the use of heterogeneous teams of human agents, Unmanned Aerial Vehicles (UAVs), and surface and ground vehicles. The intent is to provide situation awareness, services and supplies to emergency rescuers on the ground in collaboration with other human resources in disaster relief scenarios. 2 Much of the recent work with collaborative human/robotic systems [55,82,50,57,35,52,49] has focused on the representation and generation of shared tasks and shared goals and how such shared goals can be achieved through the coordination of agents participating in such shared tasks. In other work, we have considered this and developed a delegation-based framework for task generation, allocation, and execution for collaborative robotics [36,33,37]. In fact, these earlier results will be used together with the work presented in this article, for supporting collaborative data collection tasks and other types of missions. The delegation framework will be used to setup distributed data collection, distribution, and synchronization of tasks among teams of robotic and human agents. These data stores then contribute to a shared situation awareness of different aspects of the operational environment that can be used by robotic and human agents alike in other tasks associated with emergency rescue.

SymbiCloud HFKN Framework
The focus of this paper is the SymbiCloud HFKN Framework, which includes a data/knowledge management infrastructure that is intended to be used to support distributed, collaborative collection of data and knowledge and its shared use in multi-agent systems. In this framework, each agent is assumed to have a SymbiCloud module (SCModule) containing its local or contextual perspective of its operational environment. This module can include geographically tagged information, sensor-data abstractions, 3D maps, static and dynamic object representations, and activity recognition structures, in addition to a rich set of reasoning engines and data/knowledge management processes. An agent's SCModule content will vary according to its capabilities, its sensors, and its ability to gather information. An agent's data/knowledge perspective can be enhanced and extended through interaction and synchronization with other agent's data/knowledge perspectives. These shared perspectives provide enhanced situation awareness for individual team members.
Given a team of agents, information in SCModules can be aggregated and merged dynamically and virtually at different abstraction levels to allow for richer perspectives to improve timely decision making and planning processes for the individual agents and teams. The Symbi in SymbiCloud is intended to emphasize that we would like to semantically ground as much data and knowledge as possible. The Cloud in SymbiCloud is intended to emphasize the highly distributed and fluid use of data and knowledge across individual team members and the goal of leveraging developments in Cloud-based technology in a wider perspective, where parts of an agent's SymbiCloud module may be stored and accessed using Cloud services. Additionally, agents with SCModules can access useful and relevant information directly from the Internet such as Semantic Web content and combine it with an agent's locally stored knowledge. Figure 1 provides a high-level schematic of the basic data/knowledge functionality we are interested in providing for teams of human/robotic agents. In this figure, there are four agents, one human agent, one Unmanned Surface Vehicle (USV) and two Unmanned Aerial Vehicles (UAVs). Each agent has the functionality to generate and share collaborative tasks through the use of a distributed delegation framework which is part of the HFKN Framework, in addition to generating, synchronizing, querying and sharing data/knowledge through the use of the distributed HFKN Framework.
For the purposes of this paper, one can abstract the distributed knowledge network from the high-level schematic depicted in Figure 1. This aspect of the framework is shown in Figure 2.
Each agent in the system is assumed to have associated with it, a SCModule. Additionally, it is assumed that each robotic agent is ROS [68] compatible 3 . Each SCModule stores the local or contextual situation awareness of an agent, structuring associated data and knowledge at different levels of abstraction. An SCModule can in part, be viewed as a generalization of a layered Geographical Information System (GIS), but with a much richer variety of data and knowledge structures and more general querying mechanisms to access information in SCModules. Additionally, SCModules include a KDB Manager for synchronizing and merging information and knowledge content across agents.
Robotic and human agents are intended to have access to data, information and knowledge at many different levels of abstraction ranging from low-level sensor data, such as point data in single scans from LIDAR (Laser Imaging Detection And Ranging sensor), or collections of images from camera sensors. Intermediate levels of data and knowledge may contain 3D or infrared maps, that are the result of post-processing of lowlevel sensor data. These structures may in turn be semantically labeled with identifiable geolocated objects and additional semantic properties. Relations between such objects may then be defined and information stored about both static and dynamic activities of such objects. High-level semantic representations provide Figure 1: High-level schematic -the collaborative robotics framework assumes that each participating agent has a Delegation Module and a SymbiCloud Module. Agents on a team generally communicate via speech acts. Each agent is assumed to be ROS compatible and leverages ROS middleware functionality while generating distributed tasks and distributed situation awareness.
qualitative models possibly grounded in lower-levels of the knowledge and data abstractions associated with an agent.
In actual emergency rescue scenarios, there are a number of important contingencies that arise that make the design and implementation of the functionalities included in the HFKN Framework relatively complex: • Unreliable communication between agent systems.
• Out-of-range issues between agent systems.
• Agent systems entering and exiting operational environments dynamically.
• Agent systems turning off for such activities as recharging or refueling.
• Queries that return no data, or only partial data in the context of data required by the querying agent. Such contingencies present an additional level of complexity in designing the functionalities of interest. These issues must be dealt with in order to build robustness and resiliency into the framework design and use. Some of these issues can be dealt with by leveraging existing communication functionality in middleware such as ROS/ROS2, but other aspects must be taken into account in the associated algorithms as will be shown.

Contributions and Content
The paper includes the following contributions: • A description of a general system architecture (SymbiCloud HFKN Framework) for supporting multimodal data/knowledge storage, in addition to the dynamic aggregation, sharing, transfer and querying of such data/knowledge in multi-agent contexts consisting of human and robotic agents. Information ranges from low-level sensor data collected by robotic sensors to high-level semantic knowledge.
• An integration of an existing delegation-based multi-agent framework with the HFKN Framework that together is used for automatic generation and execution of collaborative data/knowledge-collection tasks.
• A synchronization algorithm and protocols which allows for the automatic distribution and sharing of information and knowledge between agents, through the synchronization of RDF (Resource Description Framework) Documents/Graphs.
• A data-transfer algorithm and protocols for exchanging datasets of low-level sensor data among agents based on the use of metadata about such datasets.
• An empirical evaluation of the HFKN Framework and associated algorithms.
• A field robotics emergency rescue case study based on a multi-agent data collection mission that uses all described functionalities of the HFKN Framework.
The paper is structured as follows. In Section 1, the basic context for this work in addition to a conceptual description of the HFKN Framework has been introduced. In Section 2, a brief summary of the Delegation Framework used by the HFKN Framework is described. In Section 3, the structure and content of SCModules, that provide the data and knowledge content of agents is presented. In Section 4, the concept of a dataset which specifies a collection of sensor data in addition to its metadata description is provided. In Section 5, the main algorithm and protocol for distributed data and knowledge synchronization used by the HFKN Framework is presented. In Section 6, the processes as to how agents exchange low-level sensor data via a dataset transfer protocol is presented. In Section 7, a number of validation experiments related to the synchronization and dataset transfer protocols are provided. In Section 8, a field robotics case study with collaborating UAVs and human agents that displays the power of the HFKN Framework in actual robotic scenarios is described. Section 9 and Section 10 provide a description of related work and conclusions, respectively. The paper also includes two appendices that provide detailed descriptions of schemas used in SCModules and examples of complex SPARQL queries that can be expressed by the system.

Brief Summary of the Delegation Framework
In the introduction, we provided an overview of the HFKN Framework which is the focus of this article. This component is part of a larger framework for collaborative robotics which leverages the use of a delegation framework [38,33,37,48] for generating and executing complex, multi-agent distributed plans and tasks.
An overview of the full architecture that combines the two is shown in Figure 1. In order to understand how the HFKN Framework leverages the delegation framework, it is important to have a cursory understanding of this framework. This section is intended to do that.
As in the case of the distributed knowledge network, one can abstract an instance of the distributed delegation network, from the high-level schematic depicted in Figure 1. This aspect of the framework is shown in Figure 3.
In the delegation framework, each member of a collaborative team is assumed to have a Delegation Module associated with it. An agent's Delegation Module contains a Delegation Manager that manages the external interactions with other agents on the team, in addition to internally managing the generation and execution of composite tasks [36]. Figure 4 provides a high-level characterization of the internal architecture of a Delegation Module.
Given a high-level mission goal specification, provided by a member of the team, the purpose of the delegation framework is to dynamically generate a task specification (often distributed among agents) to achieve the goal. This task specification often involves the use of a subset of members of the team. The task specification is generated recursively through a process where participating team members agree to do a part of the mission if they have the required resources and are able to commit to doing that part of the task specification. Each team member has the ability to broadcast for help in achieving sub-tasks associated with the larger mission specification. If successful, the net result of the process is the generation of a task specification where different parts of the specification are allocated to appropriate members of the team.
Tasks are represented as Task Specification Trees (TST) [34,36]. A TST consists of a set of control nodes and a set of elementary actions (leaf nodes) provided by participating members of the team. For simplicity, control nodes can be viewed as standard forms of control in programming languages such as sequence, concurrency, if-then, etc. During the delegation process, a tree is constructed recursively through calls to participating agents where they contribute to the overall task by providing sub-trees they are able to commit to and execute.
It is assumed that each agent publicly commits to a set of elementary and composite actions that can be used in the collaborative delegation process. In the case of UAVs, examples of elementary actions would be such actions as FlyTo, TakeOff, or Land. Composite actions might consist of more complex activities such as scanning of a region, that although internally complex for the agent, are viewed externally as elementary actions that can be used by the team to generate more complex task specifications collectively. Figure 5 depicts an example of a TST generated for a concurrent scanning mission consisting of two UAV agents.
Each Delegation Module as shown in Figure 4 consists of four conceptual components:

Inter-Agent Communication
Base System Agent System • Delegation Manager -It provides inter-agent communication to other members of the team during the delegation process. Internally, it accesses the TST Factory to generate TST nodes during the TST generation phase and the TST Executor factory to execute TSTs during the execution phase.
• TST Factory -It has the ability to generate TST nodes and TST sub-trees during the TST generation phase in the delegation process.
• TST Executor -Associated with each elementary or composite action publicly declared by an agent, is a platform dependent executor that interfaces to an agents internal functionality. The TST executor is responsible for interfacing to and managing the execution of executors associated with elementary or composite actions for a specific platform. If a TST node is a goal node type, the TST executor also has the possibility to interface with an automated planner associated with a platform to generate a sub-tree from the planner that can then be used by the TST factory.
• Constraint Server -TST nodes can contain constraints that are inherited as the delegation process progresses. In order for an agent to answer the question "can I do this? " when it receives a request from another agent, it autonomously sets up a constraint problem and checks the problem for consistency, possibly returning specific variable bindings. The constraint server handles this part of the generation process. For instance, constraints can be temporal, resource based, or associated with sensor capability.
In the context of the HFKN Framework, the delegation functionality will be used to generate and execute distributed information gathering and manipulation tasks for teams of collaborative humans and robots. Examples of information gathering and manipulating TSTs will be described in the field study, presented in Section 8.

SymbiCloud Modules
SCModules are associated with each agent and store combinations of data, information and knowledge. Additionally, SCModules are responsible for sharing, synchronizing and aggregating data, information and knowledge between agents.
We characterize three types of data (in the general sense) loosely coupled to the concepts of data, information and knowledge that are acquired, stored, aggregated and shared by agents (see Figure 6):

Elementary tasks
to be executed Control nodes provide structure

Nodes can provide mission constraints:
Must finish in the specified time Scan resolution must be at least x pixels/m 2 Figure 5: TST generating a concurrent scan by a fixed-wing and a rotor-based system. A Test-If node first checks if the rotor-based system is on the ground or in the air. From the agent's perspective, a Scan-Ground-Single task is a composite action, while from the delegation perspective it is an elementary action.
• Low-level sensor data characterizes raw output from diverse sensors. Examples include data in images from camera sensors or point data in single scans from LIDAR systems.
• Intermediate data characterizes processed raw data with minimal to moderate structure. Representation of such data are often collections of feature/value pairs stored in tables but could also be the output of sensor fusion algorithms such as point clouds and 3D maps.
• High-level semantic data characterizes data generally structured around objects with properties and relations between objects. Such data is normally semantically grounded. Common examples would be ontological structures, logical structures, or graph-structured knowledge. Objects representing humans identified in search and rescue missions with feature data associated with each human is a good example.
These different types of data pose different requirements on storage, synchronization and query mechanisms due to each having different characteristics. Low-level sensor data is typically very well structured and normally acquired in high volumes with high frequency. It is important that efficient data structures are used for storage and retrieval. This data is typically retrieved with simple queries by requesting sets of ordered and timestamped frames. High-level semantic data, on the other hand, requires low-volume for storage and is normally generated with low frequency, but it requires more complex query mechanisms. High-level semantic knowledge require structures and representations suitable for distributed querying, merging and synchronization. Figure 6 depicts this informatic situation for several agents.
Each SCModule as shown in Figure 7 consists of three conceptual components: • A PostgreSQL database -This database generally stores low-level sensor data and intermediate data, in addition to any table-based information used by the agent.
• A repository of RDF Documents/Graphs -This RDF Document/Graph repository includes both object-level semantic content in addition to metadata representations of low-level sensor data and intermediate-level data stored in the PostgreSQL database. It is these structures that will be shared and synchronized between agents using the Knowledge Database Manager.
• A Knowledge Database (KDB) Manager -The KDB Manager is responsible for the management of the PostgreSQL database and the RDF Document/Graph repository. It also supports communication with other agents and handles dataset generation processes, in addition to the execution of synchronization and dataset transfer algorithms between agents.
In order to represent high-level semantic information, SCModule's use the Resource Description Framework (RDF) [53]. RDF provides a representational foundation for the Semantic Web and was developed within the World Wide Web Consortium (W3C). The RDF framework has a very simple, but powerful set of representational primitives. Most important are RDF Triples consisting of a subject, predicate and object. Subject and object are nodes in an RDF Triple and predicate is a label for the edge connecting them that expresses a relationship between the subject and the object. Subject and predicate are specified using Internationalized Resource Identifiers [39] (IRI's), that provide a unique identity to any subject or predicate, while object can have any value including an IRI.
It is useful to view RDF Triples graphically (see Figure 8), since collections of such interconnected triples can be conceptually viewed as RDF (knowledge) Graphs. Logically, an RDF Triple (subject, predicate, object) corresponds to an atomic formula, predicate(subject, object). This correspondence is very powerful since it provides formal semantic interpretation of collections of RDF Triples as logical theories. There are many extensions to the RDF specification such as RDF Schema (RDFS) [16] and the Web Ontology Language OWL [59] which extends RDFS and is used for specification and reasoning about ontologies.
Although a collection of RDF Triples can be viewed as an RDF Graph, such graphs are encoded as RDF Documents. Consequently, we use the terms RDF Document and RDF Graph more or less interchangeably in the paper. Later, we will define an extended version of an RDF Document, denoted RDF ⊕ Document, that is given as input to the RDF Graph synchronization algorithms used to synchronize shared RDF Documents between agents. A common way to store and access content in an RDF Graph is to store the triples in a SQL database, create appropriate RDF Views for the content, and then use SPARQL (SPARQL Protocol and RDF Query Language) [74] together with these RDF Views to query RDF Graphs. This is the approach we take. The KDB Manager in an agent's SCModule provides the ability for the agent, and other external agents, to access and query RDF Graphs through the use of SPARQL. From an implementational perspective, all RDF Graphs are represented in table form in the PostgreSQL database. In this case, the KDB Manager has a library of views (schemas) created using RDF View [71]. RDF View provides a means of automatically converting any SQL table into a set of RDF Triples. The Sparqlify language [78] is used to specify RDF Views.
Low-level sensor data and intermediate data is stored in SQL tables in order to efficiently support high volume and frequency data storage and retrieval via use of a PostgreSQL database and its associated SQL query language. Theoretically, one could translate any SQL tables into RDF Graphs, but it has been shown that SQL databases have on average between 5 to 10 times higher performance than SPARQL databases [11]. This is one reason we retain a conceptual separation between low-and intermediate-level data, with high-level semantic data in the KDB.
Although an agent can also access raw sensor data in the PostgreSQL database via the KDB Manager

High-level
Low-level Sensor data Semantic data Intermediate Qu er y Re sul t R e q u e s t d a t a T r a n s fe r d a t a Low frequency Low volume High frequency High volume Figure 6: In a multi-agent system each agent is equipped with its own SCModule providing its particular perspective of the world. To participate in collaborative missions it is often necessary for agents to access information from other agents' SCModule such as query for knowledge stored or request the transfer of data.

Subject Object
Predicate/Property using SQL (or SPARQL, if the requisite RDF Views have been created), such direct querying of raw data has limited usage in the types of missions we envision. Instead, collections of raw data are bundled as a dataset and information about its contents is represented as metadata using RDF Graphs. Metadata in this context is high-level semantic information about a collection of sensor data acquired by an agent during an information gathering mission. Agents query metadata about respective datasets in order to determine what raw sensor data a team of agents has collected. Datasets represent only a subset of information an agent can have, but this form of information and its conceptualization as a dataset, is practical and efficient in data-collection missions. Each agent has the ability to declare RDF Documents in its KDB as public or private. Only the public information will be accessible to other agents for sharing. One could imagine more fine-grained dynamic approaches to access-control, such as mission-based or agent role-based access control, but this is left for future work.
A more query-centric and implementational view of the SCModule is shown in Figure 9. All raw sensor data, intermediate-and high-level semantic data resides in the SQL Database (PostgreSQL). The low-level data is stored using tables with a schema specific to the type of data, while the high-level symbolic data is stored in tables with a schema suited for storing RDF Triples. These components are visualized with blue boxes in the figure.
The KDB provides four types of APIs to access the data stored in the SQL Database (orange boxes in the figure). First, a high-level mapping between tables and programming language objects is provided using Active Record [44]. Second, the data in tables can be queried directly using SQL. The two remaining APIs are compatible with Semantic Web technologies. Namely, SPARQL in combination with RDF Views and RDF/XML, Turtle (or other formats) can be used to access data represented as RDF Triples in the SQL Database. These interfaces are used by the KDB Manager, user interfaces or external Semantic Web frameworks. In an SCModule, the application layer and the query interface for the KDB are handled by the KDB Manager.

Datasets
Our focus in this paper is on complex data collection missions using multiple robotic systems with human interaction. Consequently, we specify a special structure, a dataset 4 , that is the result of any data collection mission. Additionally, agents usually require specific subsets of raw or intermediate data specified as datasets for decision making processes during mission operation. For example, an agent may want to access part of a 3D map or an orthophoto associated with a specific region in an operational environment. Consequently, datasets should be able to be created dynamically as subsets or aggregations of already existing datasets.
The description of a dataset is characterized by high-level semantic information which describes, references and annotates low-or intermediate-level data structures. This metadata description is stored as an RDF Graph. Large data, such as point clouds, images, etc., cannot generally be exchanged efficiently between agents at periodic intervals, but metadata about datasets can be efficiently exchanged.
Raw sensor data collections are grouped in datasets consisting of the actual raw sensor data that is stored in a PostgreSQL database and the associated metadata represented as an RDF Graph. In the synchronization processes managed by the KDB Manager, it is the metadata associated with datasets that is synchronised between agent platforms. When an agent does need to access and acquire actual raw sensor data from another agent, the agent would need to make an explicit request that the sensor data should be downloaded from another agent using the Dataset Transfer Protocol presented in Section 6.

Dataset Representation
A dataset D consists of a set of data points {d i }. Data points can be: • Raw sensor data, such as images and image sequences, LIDAR scans, IMU frame sequences, GPS positions, robot poses, etc.
• Results of sensor fusion algorithms that combine different types of raw data, such as a sequence of robot poses that use both IMU and GPS data, or the locations of humans or building structures, derived from complex classification algorithms.  • Any other high-volume data acquired from external services, such as systems based on Semantic Web or GIS technologies.
Data points are stored in database tables according to their types. The tables contain a set of fields that are specific to the type (for instance, IMU contains fields for linear acceleration and angular velocity, as well as covariances). All data points are associated with a Uniform Resource Identifier (URI) defining the dataset they belong to. Associated with the dataset URI are additional properties about the dataset. Dataset descriptions are defined using a collection of RDF Triples: • an URI of a dataset, for example ex:dataset1.
• a geometry corresponding to the area covered by the dataset, for example: (ex:dataset1, geo:hasGeometry, "POLYGON((...))") • a type indicating the type of data, for example, image, LIDAR scan, victims (salient points representing potential victim locations): (ex:dataset1, aiics:dataset type, aiics:points cloud) • a list (possibly empty) of other datasets (URIs) included in this dataset. This is useful to allow the modular structuring of a dataset into smaller component datasets: (ex:dataset1, aiics:dataset include, ex:dataset1) (ex:dataset1, aiics:dataset include, ex:dataset2) Figure 10 shows an example RDF Graph representing metadata for two point cloud datasets, where one is a subset of the other. For more examples of sensor data representations, see Appendix A.

Dataset Relation to Agents
Given a team of agents, each agent stores a collection of RDF Graphs as part of the KDB in its SCModule. The union of collections of these RDF Graphs virtually represents the global knowledge of a team of heterogeneous agents. Since, the data/knowledge is distributed in multiple databases and owned by different agents, no agent has direct local access to the complete spectrum of information associated with a team. In order to gain access to the required information necessary for achieving a specific mission goal (e.g. a  Figure 11: An example of a set of datasets and the relationship between them. There are three datasets (in blue) resulting from an exploration mission, created by two UAVs. A point cloud dataset (in orange) was created by fusing two LIDAR scan datasets. A building dataset (in green) was created by processing the point cloud and an image dataset. The graph shows how each dataset was created and also which agent currently holds a copy of a dataset.
global map of the operational environment), the agents have to collaborate to fully (or partially) synchronize their data/knowledge resources, or to eventually exchange raw sensor data. The synchronization process is intended to be fully automated. It occurs asynchronously, in particular when a team member updates a public RDF Document subscribed to by other agents.
Each agent is identified uniquely by its own URI. Datasets are coupled to specific agents by the creation of RDF Triples that relate the unique agent URI to datasets (also stored in RDF Graphs) via the following relations: • has -which indicates that an agent has a copy of the raw data associated with the dataset in its database.
• created by -which indicates which agent has acquired or processed raw data.
• created from -which indicates that a dataset was created using raw data from another dataset. For instance, a point cloud can be the result of combining multiple LIDAR scans together from different agents. Figure 11 shows an example of a set of datasets owned or shared by two UAV platforms and one human operator.

Dataset Discovery
Agents are aware of their team members via their unique URIs and by leveraging identification functionality in the framework. Additionally, as agents act in operational environments and create datasets, each has an RDF Document listing those datasets and other information publicly available. A Publish-Subscribe ROS-based strategy is used to declare publicly accessible documents that can be subscribed to by any agent. An agent's RDF Document listing public datasets is published by default. Generally, an agent subscribes to all public datasets specified by this RDF Document for each team agent, although the subscription strategy can be modified by any agent at any time.
This team awareness implies that agents have the ability to query each other locally through SPARQL interfaces managed by their KDB Managers. For instance, if one agent is interested in a specific geographical region, it can use its synchronized shared information and discover what type of information and data associated with that geographical region is available from the team. Figure 12 shows an example of a set of synchronized multiple datasets created and partially shared by a team of agents after querying each other and subscribing to the relevant RDF documents of interest (datasets).
In order to exchange the actual raw sensor data associated with a dataset as described by its metadata, the delegation framework is used with a TST (Task Specification Tree) containing two nodes for uploading and downloading raw data between the corresponding agents. In the Delegation Framework, a TST is both declarative and executable, so such TSTs essentially drive the execution of dataset transfer processes. Downloading is executed on the agent receiving the data, where it first sets up the proper data storage capability. Uploading is executed on the agent sending data, where it extracts the specific data in its KDB and then sends it on the network to the receiving agent. The Dataset Transfer Protocol and processes are described in Section 6.

Datasets and Multi-Agent Data Collection Missions
In a typical collaborative exploration scenario, each agent explores part of the environment, stores data collected during a mission locally, and shares meta-information about the collected data as a dataset with other agents that subscribe to that dataset. Raw sensor data is only physically exchanged between agents if required. Some examples of when an agent would need large quantities of raw sensor data would be for visualization by a ground operator, for backup purposes, or for further processing such as extracting higherlevel semantic information from the raw data. This could involve people identification or classification of building structures which would use specialized algorithms local to an agent. Figure 13 shows a typical data collection mission evolution from the perspective of the data/knowledge that is locally available to several interacting agents. The setup involves two UAVs and one human operator. As can be seen, metadata about collected sensor data is synchronized between agents continuously while actual sensor data is exchanged only if an agent explicitly requests it. This is a by-need strategy. These two processes are considered in detail in the following two sections.

RDF Graph Synchronization
In the HFKN Framework, high-level semantic information is represented as RDF Graphs, which are encoded as RDF Documents. An RDF Document can contain one or more RDF Graphs. An RDF Dataset [29], not to be confused with our use of datasets in Section 4, is a collection of named RDF Graphs with no more than one (unnamed) default graph. Lower-level data, for example from sensors, is handled differently as described in the previous sections. This split allows for low-bandwidth synchronization of high-level knowledge and on-demand access to high-bandwidth low-level sensor data and intermediate data. This section focuses  on algorithms in the HFKN Framework that deal with achieving a common view of the agents' shared knowledge in addition to its meta-knowledge about low-level sensor data. This is done through RDF Graph synchronization, which results in a common view of a team's shared RDF Graph collection and content.
Agents use RDF Graphs to store various types of information, such as agent capabilities, list of salient objects, metadata about datasets, general knowledge useful for decision making and planning, etc. Throughout a mission, agents make changes to those RDF Graphs. For instance, while collecting sensor data and processing it, an agent will need to update the metadata of the associated dataset. Achieving a common view of a shared RDF Graph between agents by simply exchanging it in its entirety is infeasible and impractical for several reasons. First, it is inefficient when exchanging all of its content at all times. Moreover, agents may join and leave throughout the execution of a task or a mission. Communication between agents is also unreliable by nature since we deal with mobile robots and wireless communication links. This may result in partial transmissions or receiving incomplete information. Most importantly, a simple exchange of an entire set of RDF Graphs does not solve the consistency problem when multiple agents sharing the same RDF Graph make simultaneous updates. Instead of communicating the whole RDF Graph content after every change, the HFKN Framework encodes changes as deltas representing incremental changes in an RDF Graph. These deltas are then used more efficiently in the synchronization processes.
It is assumed that each agent has a dedicated RDF Document containing information about those RDF Graphs it will make publicly accessible. This information can of course be updated throughout an operational mission. As mentioned previously, a publish/subscribe mechanism is built into the system that allows agents to both publish (share) and subscribe to public documents. If agent A subscribes to an RDF Document from agent B, then that document becomes shareable by both agent A and agent B and can then be updated by both agents. Prior to this, the document can only be updated by Agent A (if there are no other agents that have already subscribed to this document). This generalizes to teams of agents collectively sharing and thus being able to update collections of RDF Graphs.
The main problem then becomes how to guarantee that this collective sharing of RDF Graphs is guaranteed to stay synchronized (consistent) across all agents involved, while updates are made locally and concurrently by these agents. As mentioned, the added difficulty is due to the fact that there are communication breakdowns among agents and that agents both enter and leave the operational environment in question. Recall also, that some of these shared documents contain meta-information (dataset representations) about large quantities of sensor data collected during data gathering missions. The synchronization processes only keep this data consistent among agents indirectly through synchronization of the RDF Graphs containing meta-information. Consequently, data-transfer algorithms are required for physically transferring low-level sensor data among agents when that is required.
Our approach is inspired by software systems used for code-versioning where new data is incremental and forms a graph in which new versions are nodes (called revisions). Incremental changes are differences between subsequent revisions (called deltas) and are the edges between revisions. This approach allows for recreating complete instances of the versioned information by starting at a specific revision, backtracking recursively through the parents to the initial revision, and finally applying all the deltas sequentially in a forward fashion. While in the traditional application, versioning is performed when needed as determined by the programmer, in the HFKN Framework, we are interested in continuous synchronization as soon as new information is available.
Traditional code-versioning systems, such as Git, encode the source code as text files, where the deltas between two different code revisions are added, deleted, and merged text lines. While the RDF Graphs could also be encoded as text, using such a representation becomes problematic. Two semantically identical graphs can be encoded in an infinite number of ways depending on the actual text format used, for example, using different orders of RDF Triples or the number of spaces between the terms. A system that uses textbased representation would essentially create new revisions based on these encoding differences even if the two versions of the RDF Graph are identical. A simple solution would be to adhere to a very strict encoding of an RDF Graph (order, spaces, etc.), but even such a solution could lead to merge conflicts that cannot be resolved automatically.
In the HFKN Framework, deltas representing differences between two RDF Graph revisions are computed at the syntactic structure level rather than at the encoding level. Deltas describe the evolution of an RDF graph in the form of added and/or deleted RDF Triples that are to be shared among all participating and interested agents in order to achieve shared, consistent, common knowledge. Using such an approach, the system can recognize the difference between two versions of an RDF Graph independently of its encoding. The basic concepts of revision management, such as merge and rebase used in the HFKN Framework, are directly inspired by those available in code-versioning systems. We have adapted both concepts to handle computing of deltas at a syntactic structure (RDF Triple) level.
Synchronization of RDF Graphs among agents in the HFKN Framework is achieved by exchanging messages with the goal of having a common and consistent view of the respective shared RDF Graphs. There are several assumptions and properties concerning participating agents, their roles, communication capabilities, and exchanged messages. The following assumptions exist regarding the operation of agents: • Agents operate asynchronously and at arbitrary speeds.
• Agents may experience failures to operate or communicate.
• Agents store data locally in permanent storages.
• Agents operate in good faith and are truthful.
• Agents use synchronized clocks.
• Agents are aware of other agents' existence based on periodically exchanged status messages. Agents can take one or two roles in the RDF Graph synchronization mechanism: • Each agent has a role of a standard agent.
• One agent has an additional role of a merge master awarded in an election process if more than two agents exist. The merge master is part of the RDF Graph synchronization mechanism and is responsible for integrating all the changes from the other agents into the common view. There can be different merge masters for different documents. Synchronizing RDF Graphs among a team of agents assuming the above-listed properties requires that each agent interested in a team's shared common knowledge, performs several tasks described in Section 5.2 and respective subsections. The different algorithms allow for synchronizing different revisions of RDF Documents and handling the exchange of necessary information required for synchronization. The responsibilities of a merge master and the process of electing the master among the agents are detailed in Section 5.2.4.
Agents cooperate by exchanging messages with the following properties: • Messages are sent asynchronously and without guarantees on arrival time bounds.
• Messages can be lost, arrive out of order, or be duplicated.
• Received messages are not corrupted.
• Agents exchange four types of messages: Status, Revisions-request, Revision, Vote.
These assumptions stem mainly from the fact that the communication between agents is achieved using wireless links, which are inherently unreliable. Moreover, partially received or corrupted messages are ignored upon detection. A detailed description of the four types of exchanged messages is presented in Section 5.1.6.
In the remainder of this section we start by defining the necessary terminology in Section 5.1, followed by the description of the method for RDF Document synchronisation among agents in Section 5.2. Detailed descriptions of two major algorithmic components of the proposed solution, namely merge and rebase algorithms, are presented in Sections 5.3 and 5.4, respectively.

Terminology and Definitions
In this section, we consider both terminology and definitions used in the RDF Graph Synchronization processes.

RDF ⊕ Document
In [29], an RDF Document is defined as an encoding of an RDF Graph. To facilitate RDF Graph synchronization, the RDF Document specification is extended and used in the HFKN Framework to also specify the incremental changes of its associated RDF Graph. This history of changes is represented using a Graph of Revisions, where vertices represent particular revisions of an RDF Graph and edges represent incremental differences (i.e. deltas) between revisions. We use the term RDF ⊕ Document for this extended specification. It includes the Graph of Revisions for the RDF Graphs in the document. Figure 14 presents a schematic of an example RDF ⊕ Document. It depicts the main terms and concepts, as well as relations between them. At any given time, there exist several RDF ⊕ Documents (here: A and B) which need to stay synchronized between interested agents. A specific RDF ⊕ Document encodes an RDF Graph and its history in the form of incremental changes expressed using RDF Deltas, which are parts of Revisions (highlighted by red dashed lines in Figure 14). In the example, there are seven RDF Graph versions created by five agents (Agent A...E) at different times during a mission execution. Different versions are connected with edges, which include RDF Deltas representing differences between them. The graph containing the revisions can exhibit a split which results in separate branches. Two example branches are marked with striped areas in the figure. Branches can undergo a merge operation when needed. The leaf nodes in the graph, that is the latest versions from the agents' perspectives, are called heads. The operations of Split, Branch and Head, associated with the construction of a Graph of Revisions, are considered in detail in Section 5.1.3.
Using an RDF ⊕ Document it is possible to instantiate all past versions of the RDF Graphs associated with it, including the latest version (i.e. head), which should be in sync between all collaborating agents that share the document. This structure allows for implementing an efficient mechanism for creating, updating, and synchronising RDF Graphs shared among a team of agents. Multiple agents can work in parallel on the same RDF Graph, and new agents can obtain the newest revision of an RDF Graph created by other agents.
An RDF ⊕ Document can be easily converted into a standard RDF Document and used in combination with other frameworks based on the use of Semantic Web technologies. Currently, our system allows for direct querying of RDF ⊕ Documents using SPARQL.
A comprehensive specification of the encoding of an RDF ⊕ Document is provided in Appendix A. The original definition of an RDF Document in [29] uses text-based encoding of an RDF Graph. In the HFKN Framework, an RDF ⊕ Document is encoded using a binary representation (see Appendix A, similar to [42]). This representation is more suitable for representing sensor data such as images as well as storing it in a database, while being compatible with text-based Semantic Web frameworks.

RDF Delta
There have been several proposals for efficiently computing the differences between two RDF/S Knowledge Bases [92] or RDF Graphs [9,91]. These proposals are based on syntactic, semantic, or combinations of syntactic and semantic characteristics of the RDF Graphs in question. Our approach is based on one of the more straightforward definitions of deltas defined in terms of a plain set-theoretic semantics based on two operations of insertion and deletion of RDF Triples. Other approaches in the literature could be used that provide more efficient deltas without changing the basic operations of the HFKN framework, but we save this for future investigation.
An RDF Delta (∆) is an expression of the difference between two individual RDF Graphs (or two versions of the same RDF Graph). Given two RDF Graphs G i and G j , the difference can be summarized with the list of inserted triples The removed and inserted lists are swapped when switching the order of graph indices i and j such that: We define a delta between G i and G j as: RDF Deltas are encoded using a subset of SPARQL Update query [74] as presented in Appendix A.

Graph of Revisions
The history of changes of an RDF ⊕ Document is represented as a Graph of Revisions (GoR). Nodes in a GoR represent revisions, and edges represent deltas. The following content defines a revision: • A list of RDF Deltas associated with parent revisions -∆ i,j , where i and j are hashes (see below) of the parent and this revision, respectively. In the HFKN Framework, each revision has one or two parents (see Figure 14). Root revision is an exception and has no parents.
• The UUID of the author of the revision.
• The timestamp corresponding to the time when the revision was created 5 .
• A hash used to identify the revision, and computed over its content using the SHA-512 algorithm [81] as follows 6 : • A cryptographic signature computed over the hash using the RSA algorithm [7].
The hash allows for the unique identification of a specific revision as it is computed over all of its contents. The cryptographic signature has no direct use in the current version of the HFKN Framework since we assume that all agents are trustworthy. Nevertheless, it is included in preparation for extending the framework to handle the trust level between agents planned in future work.
The root revision node in any GoR is defined as a null revision. Specifically, the list of RDF Deltas is empty (i.e. it has no parents), and the values of the author's UUID, timestamp, and signature are null. Published revisions are immutable, meaning that the deltas, parents, author, and timestamp cannot change after publication. Before publication, revisions can be moved to a different branch, which means changing the hash and parent of a revision.
A head is a revision that does not have any children. Figure 14 shows an example GoR with two heads: G 5 and G 6 . The set of heads is denoted as H. A split is defined as a revision that has two or more children (e.g. G 0 and G 2 in Figure 14). A branch is defined as a set of revisions between a head and a split. The branch includes the head but excludes the split. An example GoR presented in Figure 14 contains two branches If all the revisions are sequential, the GoR is linear. An example of such a case would be the GoR presented in Figure 14 if we exclude G 2 , G 4 and G 5 revisions. Whenever RDF ⊕ Document changes occur concurrently (e.g., performed by two different agents), the resulting revisions have to be combined into a single consistent one. This can be done using two different approaches: merge or rebase. In Figure 14 a result of a merge between two revisions G 3 and G 4 results in a new head revision G 6 . More details about the merge and rebase procedures are presented in the following subsection.

Combining Branches: Merge / Rebase Overview
Whenever an agent makes an update to an RDF ⊕ Document by adding or removing RDF Triples to the encoded RDF Graph, it computes a delta and creates a new version of the RDF ⊕ Document. It then broadcasts the delta to other agents that have subscribed to that RDF ⊕ Document. The delta can then be applied to the receiving agents' copy of the RDF ⊕ Document to maintain knowledge synchronization. This process is straightforward if only one agent performs document changes.
However, agents may create changes to one RDF ⊕ Document concurrently, either because they make a change simultaneously or because they are out of communication range. When this happens, the document will have several branches (i.e. heads) that need to be combined into a single one. This is depicted in Figure 14 as a split and merge.
In this work, we propose two approaches for combining RDF Graph branches: merge, which creates a new combined revision based on deltas of two other revisions, and rebase, which moves the revisions from one branch to another. Merge is more general and is always applicable, while rebase can only be applied if the revisions are still local, which means they have not been broadcasted yet.
Illustrative examples of combining revisions using the two proposed techniques are presented in Figure 15, where three agents: A, B, and C, create new revisions at different time points. An example scenario considering a merge procedure is depicted in Figures 15a and 15b. Agent A is responsible for performing the merge operation. At first (i.e. time t − 1), the document contains the triples T 0 , T 1 , T 2 , which correspond to revision G 0 = {T 0 , T 1 , T 2 }. Next, at time t, Agent B adds triples T 3 and T 4 and removes T 0 and T 1 . This constitutes revision G 1 , Concurrently, at time t, Agent C adds triples T 4 and T 5 and removes T 1 and T 2 . This constitutes revision The resulting graph of revisions is shown in Figure 15a.
Finally, at time t + 1, Agent A receives the deltas (I 0→1 , R 0→1 ) and (I 0→2 , R 0→2 ) from Agent B and Agent C, respectively. It then creates a single revision G 3 = {T 3 , T 4 , T 5 }, with two deltas by applying the merge operation: The structure of the resulting revision G 3 is shown in Figure 15b. The alternative rebase process is illustrated in Figure 15c. At time t, Agent C creates revision G 2 locally (i.e. does not publish it). Next, at time t + 1, the agent receives a new revision G 1 from Agent B. Agent C then can rebase G 2 on top of G 1 and create a revision such that I 1→2 = {T 5 } R 1→2 = {T 2 }. The structure of the revisions is shown in Figure 15c.
In summary, the HFKN Framework uses two techniques for combining changes in RDF ⊕ Documents: • merge combines two branches by creating a new revision with two RDF Deltas. The advantage of merge is that it can be applied to any branches, and the new revision can be synchronized with other agents. The drawbacks are that it creates a new revision, which makes the revision graph more complex. Additionally, if only merge is used, and it is not performed in a timely fashion, the global synchronization between all agents may not be guaranteed in certain scenarios due to practical limitations (discussed in Section 5.1.5).
• rebase moves a branch on top of another branch, resulting in a single branch with all the revisions. The advantage is that it reduces the complexity of the revision graph and minimizes the number of revisions. In the HFKN Framework we limit the use of rebase only to local revisions to avoid the need for additional functionalities for guaranteeing that identical rebase operations are performed by all agents. Further details of merge and rebase algorithms are presented in Sections 5.3 and 5.4, respectively. The HFKN Framework uses a combination of merge and rebase operations in order to guarantee a global synchronization of RDF ⊕ Documents among all agents. Detailed discussion of the interactions between the two algorithms and their use is provided in the following subsection.

Synchronization Protocol
Merge and rebase are two possible approaches for combining the changes in the RDF Graph made by different agents. The Synchronization Protocol describes how these two approaches are used to achieve a common view of an RDF ⊕ Document.
A naive approach in which all agents individually apply only merge operations as soon as they have more than one head in their GoRs would result in a never-ending merge loop creating an infinite number of new merge revisions. To solve this problem, distributed code-versioning systems rely on a central authority. Whenever a participant has a new version of their code that should be shared with other participants, the participant makes sure they are synchronized and consistent with the central authority.
This approach alone only works if the rate of changes of the RDF ⊕ Document is low, as each agent has to have enough time to ensure its GoR is consistent with the central authority 7 . In the HFKN Framework, agents can make changes at a much higher rate, up to several times per second. Agents may not have time to ensure consistency of their GoR with the central authority before publishing their changes. Thus, in the HFKN Framework, agents publish their changes without ensuring consistency, and merging the changes is the responsibility of a central authority. We call the central authority the Merge Master. Although, the concept of central authority is directly inspired by distributed code-versioning systems, its use in our framework is novel. In the code-versioning systems, the central authority is static, while in the HFKN Framework, the role of the merge master is assigned dynamically in the process of election (Section 5.2.4). Each agent can be selected as the merge master at any time and the election is performed among all agents within communication range. This is done for redundancy and reliability reasons to deal with dynamic mission scenarios where agents join and leave missions, as well as to deal with limitations of wireless communication links, such as disconnections.
An overview of the synchronization protocol is shown in Figure 16. When an agent makes a change, it broadcasts its new revision. When the merge master receives the new revision, it checks if the merge operation can be performed. This is done by ensuring that its GoR has all parent revisions of the newly received one. If it does not, the merge master sends a request to other agents for all missing revisions. Once the merge master has received all the revisions, it can then compute a merge revision by applying the merge algorithm. The resulting new merged revision is then broadcasted to other agents. When an agent receives the merge revision, it checks if its GoR has all the parent revisions so that the merge revision can be directly incorporated. In case any revisions are missing from its GoR, the agent will send a request for the missing revisions.
If agents create new revisions too quickly in parallel, there is a risk that the merge master cannot cope with the load. If that happens, the graph of revisions for each agent would include several branches which are never combined. Agents would only be able to make changes in their respective branches and a common view of the RDF Graph with all information from other agents would never be achieved. We call this the Never Synchronized Problem.
The Never Synchronized Problem. Let us consider the following hypothetical scenario which includes two agents: A and B. Agent A is the merge master. Both agents create a new revision every 100ms. It takes 10ms to transmit revisions between agents and 85ms for A to complete a merge.
The timeline of operations shown in Figure 17a is as follows: • At t = 0ms, Agent A creates revision R A 0 and Agent B creates R B 0 . The revisions are published. • At t = 10ms, Agent A receives R B 0 and initiates a merge.
As can be seen in this example, Agent A gets access to the changes of Agent B, but Agent B does not receive the merge information in time before creating its new revision. This results in Agent B not being able to incorporate the RDF knowledge graph changes created by Agent A. Rebase local changes and Merge public revisions. To avoid the Never Synchronized Problem, agents store local revisions for all the changes made until their GoRs are synchronized with the merge master. The following procedure, which interleaves the use of merge and rebase algorithms, is used.
• When an agent makes a change in an RDF Graph: -The agent checks if its current revision inherits from the current revision of the merge master (it is known based on the status message). If that is the case, the agent creates a new public revision and publishes it.
-Otherwise, the agent creates a new local revision (i.e. the revision is not published).
The details of this behavior are described in Section 5.2.1.
• When an agent receives a merge revision and if it has local revisions, the agent will apply the rebase algorithm using the local revisions and the latest available merge revision. This will result in a new revision that combines all changes that will be published. This is described in more detail in Section 5.2.2.
The timeline for the new behavior of the two agents is shown in Figure 17b. As we can see, both Agents A and B can make their local changes and access the other agent's changes. The timeline of operations now becomes as follows: • At t = 0ms, Agent A creates revision R A 0 and Agent B creates R B 0 . They believe they are synchronized and publish their revisions.
• At t = 10ms, Agent A receives R B 0 and initiates a merge. At that point, Agent A and B believe they are not synchronized anymore.
• At t = 95ms, Agent A complete the merge revision R AB 0 between R A 0 and R B 0 . At that point, Agent A has access to R A 0 and R B 0 . • At t = 100ms, Agent A creates revision R A 1 and Agent B creates a local revision R B 1 .
1 and initiates a merge. • ... By combining the merge and rebase methods for managing branches in GoRs, it is possible to maintain the RDF Graph synchronization between agents. Further details of the two methods and their relations are described in Section 5.2.

Messages
All communication between agents allowing for maintaining a synchronized view of the RDF Graph is realized using the following messages: • The Status message is used to discover agents subscribed to a document and that are within the communication range. It contains meta-information about the agent: its name, UUID, the public RSA key, and the state of the document. It includes the head revision hash and whether the agent believes and acts as a merge master.
• The Revision message is used to exchange a revision between agents. It contains all the information needed to insert a new revision in the Graph of Revisions: author's agent UUID, revision delta(s), timestamp, revision hash (Equation (7)), and a cryptographic signature. • The Revision-request message is used to request a revision that is not available in an agent's Graph of Revisions. The message contains the requester's UUID and a list of the required revision hashes.
• The Vote message is used during the election of the merge master. It contains the voter's UUID, candidate's UUID, the election round number, the vote timestamp and the election timestamp.
A status message is sent periodically by all agents and has three purposes. First, it allows for detecting availability of other agents. Second, it allows the merge master to know other agent's head revisions and thus know whether it has all the necessary revisions before performing the merge operation. Third, it allows for knowing which agent has the role of the merge master and if an election is necessary (i.e. no or multiple masters are present).
A revision message is sent when a change to an RDF ⊕ Document is to be communicated to other agents or upon an explicit request.
When an agent is aware of the existence of a revision through a status message or a parent of a known revision, it can use the revision-request message to obtain the missing information.
A vote message allows for initiating a merge master election process and casting votes. More information is presented in Section 5.2.4.

RDF ⊕ Document Synchronization
As outlined in previous sections, agents acquire new knowledge throughout a mission execution, and its high-level description is represented as RDF Graphs encoded with their history in RDF ⊕ Documents. To achieve a common view of the continuously updated RDF ⊕ Documents among all participating agents, the HFKN Framework uses a number of algorithms and protocols that allow for efficient and robust RDF ⊕ Document synchronization.
A schematic view of the processes, functionalities, and the exchanged messages is presented in Figure 18. The view focuses on the perspective of one agent, which hosts several processes such as a Delegation or Scanning Modules and, most importantly, from the perspective of this work, the KDB Manager. The manager receives data from these modules and updates its RDF ⊕ Documents. For each document, the agent uses several processes (in gray) to handle RDF ⊕ Document synchronization as well as communication with other agents by exchanging messages (in white) to maintain the common view of the RDF ⊕ Document.
The following processes are used to achieve knowledge synchronization. Updates reflect the new information obtained by the agent itself to the RDF ⊕ Document by creating a new local revision. The revision is either directly broadcasted to other agents or kept local. This decision is made by the HandleLocalRevisions process. Upon receiving revisions from other agents, the HandleExternalRevisions process is responsible for incorporating the new information into its version of the RDF ⊕ Document. The HandleRevisionRequest process is responsible for providing revisions when other agents request them. The periodic sending of the status information as well as receiving the information from others is the responsibility of HandleStatus process. Finally, initiating and participating in the merge master election is handled by the HandleElection process. The processes and the algorithms they employ are described in the following subsections.

Handling Local Revisions
When an agent performs its tasks, the operation results often require updating the information contained in RDF ⊕ Documents. Examples of this include obtaining new sensor data such as images or LIDAR data, specifying the agent's capabilities, battery level, to name a few. When this need arises, a new revision is created. The decision whether it should be directly published to other agents is taken according to Algorithm 1.

Algorithm 1: HandleLocalRevisions
The algorithm checks if the latest merge master's revision (G mergemaster ), received from the merge master's status message, is one of the ancestors of the new revision (G i ). The list of ancestors is computed by recursively listing all the parents of a given revision. If that is the case, it means the current changes are ahead of the merge master, and the new revision can be published to other agents. Otherwise, G i is added to a list of local revisions and will ultimately be published when the latest revision is received from the merge master, that is when they become synchronized.

Handling External Revisions
When an external revision is received from another agent, it is incorporated into the appropriate RDF ⊕ Document through an update to its Graph of Revisions (GoR). The process is performed according to Algorithm 2.
Upon receiving of a new revision, G new , the agent verifies that its GoR contains the parent revision(s) of G new . If it does not, it requests that revision. If the revision is not already in the GoR, it is then inserted.
The algorithm continues in line 6 where the current head revision G is obtained followed by handling any local revisions in line 7. If G new is a merge revision (i.e. it has two RDF Deltas related to two parents) and a rebase operation can be performed (i.e. all revisions in G are local and G and G new are connected in the agent's GoR) the following steps are performed. First, revisions G and G new are combined by the rebase algorithm (Algorithm 8, Section 5.4). The resulting revision is then published to the other agents in line 8.
The algorithm continues in line 9 with the tasks of the merge master. First, the agent waits until there is a need for merging of revisions, which means that there is more than one head in the graph of revisions. The MergeRevision Algorithm 6 is applied to the first two heads in H and the result is then inserted in the graph of revisions and published using the InsertRevision and PublishRevision functions, respectively. The algorithm continues until all revisions are merged, that is the number of heads is one.

Handling Status
All agents participating in the RDF ⊕ Document synchronization process send their status information as well as receive the statuses from all other agents. The purpose of the status message as defined in Section 5.1.6 is threefold. First, it allows for detecting the existence and availability of other agents. Second, it enables the merge master to determine other agents' head revisions and thus conclude whether it has all the necessary revisions before performing the merge operation. Third, it is used to establish which agent has the role of the merge master and if an election is required (i.e. none or multiple masters present). The procedure for handling the status messages is presented in Algorithm 3. When an agent receives a status message from another agent (σ in line 2) it first updates its own status Γ and checks in line 4 if its GoR contains the revision designated by σ. If it does not, the agent requests the missing revision in line 5.
Subsequently, after incorporating all the received status messages, the agent checks whether there is a need for a new merge master election. The check happens after updating the status to ensure the system reacts quickly to an incorrect number of merge masters. First, the agent checks if its UUID is the lowest among all available agents. This is done to minimize the number of initiated elections as only the agent with the lowest UUID can start the election process. Second, in line 8 the agent checks the number of agents which consider themselves merge masters (MM) and if that number is different than one it initiates a new election in line 9. This only happens when none or more than one agent consider themselves and act as merge masters.
Finally, in line 10 the updated version of the agent's status Γ is used to publish the agent status message as defined in Section 5.1.6.

Merge Master Election
When multiple agents are connected, they select a single merge master as discussed in Section 5.1.4. An overview of the procedure each agent follows to handle an election is presented in Algorithm 4 and graphically depicted in Figure 19. When an agent starts an election process as specified in line 9 of Algorithm 3, it publishes a vote message (Algorithm 4 line 6). Similarly, when an agent receives a vote, and is not in the election mode (election = f alse), then the agent knows that a new election has started (line 4) and its vote is published in line 6. The structure of the vote message is defined in Section 5.1.6 and the vote value is determined according to the following rules: • pick the last agent for which the elector voted for to be a merge master if that agent is still connected, or • pick the agent to which the elector has been connected to for the longest time.
Each agent receives the voting information (line 7), which can then be used to determine the unique merge master in a decentralized manner. To handle unreliability in communication, agents wait for a specific duration after the start of the election before they decide on the winner (line 7).
In the case of ties, a second stage of the election is necessary (i.e. resolution), in which only the agents with the most votes are considered. In the resolution stage, the agents vote (line 10) for a random candidate, and the procedure is repeated until a unique merge master is selected. With each new round of the election, the election round number (round i ) is incremented in the Vote message.

Requesting Revisions
An agent can become aware of the existence of revisions through a status message from another agent, or by receiving a revision with parent revisions which does not exist in its GoR. In such a case, an agent can request the missing revision and complete its graph of revisions. This is performed in Algorithm 2 (line 4) or Algorithm 3 (line 5). Notice that this also happens when a new agent joins the mission or becomes reconnected after a temporary communication loss.
A simple strategy for handling responses to requests which would require an agent to respond to all requests individually (assuming it has the requested revision) would be highly inefficient. Since the requested information in many cases can be already stored by multiple agents, this would lead to sending an unnecessary amount of duplicated messages. Instead, to minimize the number of responses and exchanged messages, we propose to use the strategy presented in Algorithm 5. After an agent receives a revision request (line 2), it responds by publishing the requested revision if the following conditions hold. First, the request came from the merge master (line 3). Second, the agent is the creator of the requested revision, or the agent is not connected with the revision creator (line 4).
In case the request came from an agent other than the merge master (line 6), the agent responds only if itself is the merge master. By using Algorithm 5, the HFKN reduces the number of messages exchanged while guarantying that the requested revisions are sent.

Illustrative Examples of Agent Behaviours
To illustrate the resiliency of our mechanism, we consider a few realistic scenarios that can occur during the synchronization process implemented in the HFKN Framework.
The merge master disappears without correctly publishing the last merged revision. When that happens, the other agents will keep updating their RDF ⊕ Documents locally by creating new local revisions (Algorithm 1 line 5). Since the merge master is no longer available (e.g. due to communication failure or hardware malfunction) its status message is not broadcasted anymore. Thus all participating agents will stop receiving the merge master's status message (Algorithm 3 line 2) and the agent with the lowest UUID will then trigger an election (Algorithm 3 line 9). After the election process is completed, the new merge master will continue the synchronization process by merging the necessary branches.
There are multiple merge masters. The team of agents may become split into two (or more) groups due to communication range limitations. Each group in that case will have its own merge master elected. When communication between agents is re-established, all the agents will receive a status message from two (or more) merge masters (Algorithm 3 line 2), and the agent with the lowest UUID will trigger an election (Algorithm 3 line 9). After the election process is completed a single merge master is selected. And the synchronisation process continues normally.
A new agent joins a mission and has no previous knowledge of a particular RDF ⊕ Document. A common scenario during any mission execution may involve a team of agents collectively acquiring sensor data and updating shared RDF ⊕ Documents. When a new agent joins the team and has no previous knowledge of the particular shared RDF ⊕ Document, its graph of revisions will be empty. The new agent will receive the status message from the merge master (Algorithm 3 line 2) with the hash value of the latest revision. The new agent will then first request from the merge master the latest revision (Algorithm 3 line 5), and then recursively all the past revisions (Algorithm 2 line 4). The recursive request process will continue until the new agent has access to the entire GoR and can get the synchronized view of the shared RDF ⊕ Document.
There is no merge master. In this case, the synchronization process is suspended, which means that agents will not get the information from other agents. However, agents are still able to update their RDF ⊕ Documents locally, by creating new local revisions (Algorithm 1 line 5). In parallel, a new election is started and after it is completed, the newly selected merge master will resume the process of merging revisions (Algorithm 2 line 9).

Intermittent communication interruptions.
Consider the following scenario involving four agents: A, B, C, and D. Agents A and B form Group 0, and agents C and D form Group 1. Agents within each group are assumed to have a reliable communication. In this case, agents A and B will always receive each others status messages, and the same is true for agents C and D (Algorithm 3 line 2). Agents A and C, due to their UUID numbers are always elected as merge masters in their respective groups.
Consider a case in which the communication between Group 0 and Group 1 is being constantly interrupted, i.e. very unstable, groups are disconnected and re-connected. When communication between groups is reestablished, the agents will have two merge masters in communication range, which will trigger an election (Algorithm 3 line 9), where either agent A or C is elected. When communication between groups breaks down, one group will still have a merge master, while the other will require a new election (Algorithm 3 line 9). If the communication links get constantly interrupted, there will be a situation where merge master election process will be triggered repeatedly. This will slow down the synchronization process as merging of new changes is not performed during an election. However, the process of merging will resume immediately after the election process is completed (Algorithm 2 line 9).

Merge Algorithm
In case two concurrent changes have been made to the same RDF ⊕ Document resulting in two branches, it is necessary to combine the changes into a single revision. The merge algorithm is one of the two approaches used in the HFKN Framework that can be used for that purpose.
Consider a simple example of a GoR with two branches that contain a single change between revisions as shown in Figure 20. Given two concurrent versions: G i with the delta ∆ l,i = (I l,i , R l,i ), and G j with delta ∆ l,j = (I l,j , R l,j ) split from the common ancestor G l , the goal is to compute a new version G m with two deltas: ∆ i,m = (I i,m , R i,m ) and ∆ j,m = (I j,m , R j,m ).  In a general case G m ⊆ G i ∪ G j . However, if a triple has been removed in the branch leading to G i , that triple should not appear in G m even though it is available in G j . Therefore the new revision is defined as: Finally, the wanted deltas: ∆ i,m and ∆ j,m can be computed using Equation (8) taking into account the properties of Equation (6): where, indices i and j can be interchanged in Equations (9) and (10).
In the general case, there may be multiple revisions leading to G i and G j , and the idea is to reduce those multiple revisions into a single one and apply Equations (8) to (10). To reduce multiple revisions, we use the following combine function: The combine function is derived from: The MergeRevision algorithm is shown in Algorithm 6. The algorithm uses the CombineMany operation (Algorithm 7) which applies recursively the combine function (Equations (11) to (13)) to reduce a path of revisions into a single RDF Delta.

Complexity
The time complexity of Algorithm 6 and Algorithm 7 is considered below. Note that when integrated with the full HFKN Framework, there is additional communication and other types of overhead involved. This is considered in Section 7.

Algorithm 6: MergeRevision
Input: Complexity of a Single Merge. Assume that each revision has a maximum of n max t triple removals or additions. The distance in the graph of revisions between two RDF Graph revisions, e.g. G i and G j is denoted as d(G i , G j ). The maximum value of d is the total number of revisions, i.e. |G|. The combine(G i , G i+1 ) function in Equation (11) needs to be executed in a loop over the n max t changes in G i+1 resulting in the worst-case time complexity of: It is possible to avoid iterating over all the triples by using a hash table representation to index the triples. A hash table has a worst-case time complexity of O(n) and an average-case time complexity of O(1) [25].
The complexity of the combine function can be reduced to the following average-case time complexity (over all the possible inputs to the combine function): CombineM any(G l , G i ) (Algorithm 7) applies the combine function d(G l , G i ) times and has therefore a worst-case time complexity of: with an average-case time complexity (i.e. using hash table representation) of: M ergeRevision(G i , G j )(Algorithm 6) applies the CombineM any algorithm two times. Therefore, it has the same time complexity.
Merge K Revisions with the Same Ancestor. Assume K agents make a concurrent change to the same revision G 0 (see Section 7.1). At the k-th merge, the number of revisions is equal to |G| = k + 1. The total worst-case time complexity of the merge of the K revisions is given by: with an average-case time complexity (i.e. using hash table representation) of: General case. In the general case, when merging two RDF Graph revisions, for example G i with G j in Figure 20, the merge algorithm has to first find the shortest path between the two revisions and then apply the merge. The shortest path is found using a Bidirectional-Dijkstra algorithm [20]. In a sparse graph, the complexity of the Dijkstra algorithm is bounded by: Where |G| is the number of revisions and |E| is the number of edges between each revision. In a GoR, each revision has a maximum of 2 parents, therefore |E| ≤ 2|G|. The complexity of finding the shortest path between two RDF Graph revisions, for example G i and G j becomes: The complexity of computing the delta is given by: The worst-case time complexity of a single merge is therefore given by: As the graph grows, it is likely that log(|G|) >> (n max t ) 2 and therefore the worst-case complexity is given by Equation (25).

Rebase Algorithm
An alternative to merging revisions is the rebase operation. Rebase changes a parent of a revision, essentially moving a branch in the Graph of Revisions. In the HFKN Framework, the rebase is only applied to linear branches. A linear branch is defined as a branch that does not contain any split or merge revisions. This is not a limiting factor as changes made locally by an agent to an RDF ⊕ Document are always linear in nature. The rebase process applied in a simple example GoR with two branches is presented in Figure 21. In this scenario, two agents (A and B) created two branches in an RDF ⊕ Document. The branch created by Agent B is done locally (i.e. not broadcasted to other agents). The rebase operation applied by Agent B moves the source branch G l (created by Agent B) to the destination branch G k (created by Agent A).
The rebase process follows Algorithm 8. In line 1, the algorithm looks for the common ancestor, using the Dijkstra search. In case of the example shown in Figure 21, the common ancestor is G i . G next holds the next parent revision when moving, it is initialised in line 4 to G k , so that the revision G l is moved on G k in line 6. G next is updated to the latest created revision in line 7 (e.g. to G l for the first iteration). Finally, the local revisions that have been rebased are removed in line 9.

Algorithm 8: RebaseRevisions
Input: After applying the rebase algorithm presented above, all local revisions are combined into the final GoR, which means the history of changes is preserved. This will directly influence the time required to synchronize changes among all agents as the number of messages used in the overall synchronization mechanism is proportional to the number of newly rebased revisions. If preserving the history of all revisions is less important than how fast the shared information is synchronized among all agents, an alternative approach to the default rebase algorithm implemented in the HFKN Framework can be used. The alternative adds an operation, called squashing before the default rebase algorithm is applied. The squashing procedure combines all of the revisions in the source branch (i.e. G l to G m created by Agent B) into a single revision, thus reducing the number of revisions in the final GoR. Squashing is performed using the CombineMany operation (Algorithm 7). Performance of both variants of the rebase algorithm is evaluated in Section 7.2.

Complexity
Following the notation from Figure 21, when rebasing revision G m on G k with a common ancestor G i , the complexity of performing the rebase operation is given by: However, it is necessary to first compute the path between G m and G k , which means the complexity of rebase is also bounded by the complexity of the shortest path algorithm given by Equation (25).

Synchronization Timing and Weak Consistency
Given two RDF ⊕ documents, we say that those documents are weakly consistent relative to each other if the RDF Triples in the documents are the same. The HFKN framework does not require logical consistency. Weak consistency is guaranteed in the following sense. Each agent shares a subset of its RDF Graphs with other agents. For each subset of RDF Graphs shared by an agent, the union of those subsets is guaranteed to be weakly consistent after termination of synchronization processes associated with the synchronization algorithms subject to certain timing and computational assumptions. These assumptions are clarified in the following scenario.
Consider the following scenario in which m RDF ⊕ Documents are shared among a team of agents. The team has made n asynchronous and/or concurrent changes to each of the shared RDF Graphs and no other changes occur. Under these assumptions, the HFKN RDF ⊕ Document synchronization mechanism will ensure that the changes are synchronized among all team members within a certain time T T otal dependent on computational capabilities of each agent and the performance of communication links. The time is defined as: Elementary tasks to be executed where T S is the maximum time to select a merge master for each RDF ⊕ Document. T M (n) is the maximum time needed to merge all changes introduced by each agent to each RDF ⊕ Document to produce a consistent version of common graphs. Time T C (n) is the maximum time required to transfer necessary revisions to all agents. Time T U (n) is the maximum time required for each agent to apply the updates received from the merge master to its copy of the RDF ⊕ Document.
In a general case, for all scenarios where the asynchronous and concurrent changes to the shared RDF ⊕ Documents finish at time t, the HFKN Synchronization mechanism will ensure that all changes are synchronized among all participating agents at time t + T T otal .

Dataset Transfer Protocol
RDF Graph synchronization is used to exchange semantic information and metadata automatically between agents. Bandwidth intensive data, such as point clouds or images are not exchanged automatically. Instead, this kind of lower-level data is encapsulated in datasets and associated with metadata represented as RDF Graphs, as presented in Section 4. The metadata is synchronized between agents using the synchronization protocol described in Section 5. When an agent needs to access the actual bandwidth intensive information, a dataset transfer protocol is used. The focus of this section is on how this is achieved.
An agent is aware of the existence and the location of a dataset from the automatically synchronised RDF Graphs. The process of downloading the data for local use is achieved through the use of the delegation framework. When the need for data transfer arises, a Task Specification Tree (TST) is generated consisting of two types of nodes: • The Send node is instantiated by the agent that has the requested data and is able to provide it to others, • The Receive node is instantiated by the agents that require the data.
During the delegation process (see Section 2), which is usually initiated by the receiving agent, a sending agent is selected. As the data can be available from several agents, the criteria for choosing a sender can include constraints such as the battery level, the available bandwidth, or if the agent is expected to stay in communication range. The delegation process results in the generation of the TST shown in Figure 22. In some cases, multiple agents may want to receive the same dataset. For this reason the TST can contain multiple Receive nodes as can be seen in the figure. The data transmitted between agents is split into chunks and transferred sequentially.
An overview of the data exchange protocol is presented in Figure 23. There is only one sender that executes Algorithm 9 and one or more receivers that execute Algorithm 10.
The protocol uses the following messages: • The Ready message is used to indicate readiness to receive data. • The Data message contains a sequence number, which allows receivers to check whether they obtained all the data, and an array of bytes containing a data chunk of the dataset.
• The ResendRequest message allows receivers to request missing data chunks.
• The Error message allows participants to indicate an unrecoverable failure and abort the transfer.
• The ThrottleUp and ThrottleDown messages allow the receiver to indicate if the sender can send messages faster or slower.
• The Finished message is sent by the sender to indicate that all the data has been sent. This message contains the last sequence number, so that receivers can check they received all the data.
The agent responsible for providing data operates according to Algorithm 9. It knows the list of receivers from the TST (Figure 22), and waits for them to be ready (line 1). The sender retrieves the data from its database (line 5) and sends it to the receivers (line 9) until all the data has been sent (line 6), then it sends the finished message (line 20). The sender checks if it has received a request for missing data (line 10) and adjusts the transfer speed up (line 13) or down (line 15) if needed. If the sender receives an error message, it aborts the process (line 17). At the end of the loop, the sender waits (line 19) for the duration τ , to make sure that the receivers can handle the reception of data messages.
An agent receiving the data operates according to Algorithm 10. It starts by sending the ready message (line 1). It then executes the main loop (line 3-16) that handles the data transfer including error handling, requesting missing data etc. The loop terminates either when the agent receives all data (i.e. receiving of the finished message) or when an error occurs (line 3). The process starts with receiving of the data message (line 4). Based on the sequence number of the received message, a check is done if there is any missing data (line 5), and if that is the case the agent requests it. Before the agent inserts the received data into its database, the validity of the message is checked (line 7). In case the data is invalid the error message is send and the execution of the main loop is interrupted (line 10). The receiving agent can request the sending agent to change the rate at which data messages are sent. This is based on the amount of data messages placed in the reception queue. If the queue has more than 5 messages waiting to be processed, the receiver sends a throttle down message (line 11). Otherwise, if the queue is empty it sends a throttle up message Algorithm 9: SendDataset

Empirical Evaluation of RDF Graph Synchronization
The RDF Graph Synchronization mechanism, described in Section 5, provides the backbone for keeping distributed knowledge collected by a team of agents synchronized, up-to-date, and accessible for use by the team. In this section, we describe simulation experiments designed to evaluate the performance of this backbone. Section 8 will provide a field robotics experiment with the full HFKN Framework, using a number of deployed UAVs and human operators.
Several aspects of the synchronization mechanism are evaluated. The first set of experiments focuses on the performance of the merge algorithm used by the merge master to synchronize RDF Graphs. The algorithm's efficiency is evaluated in a simulated scenario where a large number of new revisions are created, with varying size of RDF Triple changes (additions or deletions). The experiments solely focus on measuring the timing of the algorithm excluding any extraneous factors related to communication links. The results are presented in Section 7.1. Similar evaluation was performed for the rebase algorithm with the results presented in Section 7.2.
Additionally, a more complex simulation was performed to evaluate the HFKN Framework from a more practical perspective. A scenario, in which all functionalities of the RDF Graph synchronization mechanism are used, was created. It includes multiple agents creating, changing and synchronizing multiple RDF Graphs. During the experiment network disconnections were introduced to showcase the resilience of the HFKN Framework. The evaluation results presented in Section 7.3 include an in-depth analysis of the HFKN Framework behaviour in this complex scenario that include message exchange statistics, timeline for synchronization events and the history of all revisions over the course of the experiment. Finally, a set of experiments was conducted to evaluate the scalability of the RDF Graph synchronization mechanism. In particular, the focus was to find the maximum rate at which new revisions can be created and still synchronized in practice among all agents. This rate depends on multiple factors (e.g. number of agents, number of RDF ⊕ Documents etc.) which were taken into account during the experiment design. The results of the evaluation in scenarios including missions with up to 20 agents are presented in Section 7.4.

Evaluation of the Merge Algorithm
The performance of the merge algorithm presented in Section 5.3 was evaluated in an experiment focused on the timing requirements of the merge operation itself. This means that the overhead of communication required for the RDF Graph synchronization is not included. The experimental setup can be described by the following scenario. Consider 1000 agents making concurrent changes (in form of adding or removing RDF Triples) to a single RDF Graph. The graph is to be synchronized among all agents. Each change results in a new revision of the RDF ⊕ Document, such that its GoR will contain multiple branches. The merge algorithm is used to combine newly created revisions into one merge revision which includes all the changes introduced by all agents. Figure 24 depicts the GoR after the experiment is finished. All generated revisions (G 1 . . . G 1000 ) share the same parent revision G 0 . Since the merge operation is applied to two branches at a time, it has to be applied 999 times in this scenario in order to create the final merge revision which includes all the changes (i.e. G 1999 ). In order to quantify the influence of the size of the changes applied to the RDF Graph on the performance of merge algorithm, two cases of the experiment were performed, each containing changes consisting of 10 or 100 RDF Triples. The experiment was performed on a computer with an Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz with six cores and 16GB of RAM. 8 The results of the experiment are presented in Figure 25. They confirm that the merge operation time complexity is linear in the number of revisions that need to be combined when merging (see the red curve ...

Evaluation of the Rebase Algorithm
The purpose of this experiment is to evaluate the performance of the rebase operation described in Section 5.4. As in Figure 21, we consider two agents A and B creating new revisions in parallel that correspond to two branches in the graph of revisions of the RDF ⊕ Document. The rebase source branch (top of Figure 21) created by agent B has a varying number of revisions with varying number of changes in each revision. The content of the rebase destination branch created by agent A does not influence the performance of the rebase algorithm, therefore the destination branch contains only a single revision in this experiment. Two variations were evaluated: • move all the individual revisions from the source to the destination branch (c.f. Figure 21, revisions G l and G m are kept separate). This is currently the default behaviour of the rebase operation as used in the HFKN Framework, see Algorithm 8. • combine all revisions from the source branch into a single one before applying the rebase algorithm, using the squashing operation.
The number of revisions in the source branch was varying from 1 to 40 and the number of changes (additions and removals) in each revision from 10 to 50. The experiment was performed on a single computer omitting the communication links between agents. Other aspects, such as access to the database, were considered. We measured the timing for each variant of the rebase and each variation of the number of revisions and changes. Figure 26 shows the average time it takes for the rebase algorithm for the different configurations. The rebase operation has a constant time cost of about 23ms when only one revision is rebased. Each additional revision takes 1.3ms on average, which results in 72ms for the case of rebasing 40 revisions as shown in Figure 26. Additionally, the rebase with squashing ( Figure 26b) or without (Figure 26a) has very little influence on the overall performance of the rebase operation. Squashing reduces the number of revisions in the graph, which means that the number of messages exchanged between agents is reduced. This in turn minimizes the number of messages which have to be transmitted in the overall process of the RDF Graph synchronization mechanism. Therefore performing squashing should be preferred if the quickest synchronization between all agents is the main priority. It is important to note that in the evaluation the dominating factor is the overhead caused by accessing the database to fetch the revisions. For this reason, the number of changes and whether squashing is performed has minimal influence on the overall performance of the rebase operation.

Evaluation of the RDF Graph Synchronization
The HFKN Framework uses a number of algorithms and protocols which allow for efficient and robust RDF Graph synchronization described in detail in Section 5.2. Those functionalities are part of the KDB Manager which is executed by each participating agent. Throughout a mission execution the new information, which is acquired by each agent, results in changes to the RDF Graphs shared among all agents. In order to achieve a common consistent view of the available information among agents the RDF Graphs are synchronized. Unlike the evaluations described thus far, in this subsection the evaluation of the whole RDF Graph synchronization mechanism is presented. This experiment was designed to demonstrate the ability of the HFKN Framework to handle concurrent changes in an RDF Graph shared among agents and robustness against communication interruptions. A typical data-gathering mission is simulated where twelve agents explore an environment and new datasets are created and exchanged. Note that the focus is put only on how this affects modifications to the shared RDF Graph. Thus for this experiment, sensor readings were not simulated, and no sensor data was transferred between agents. Only the RDF Graph synchronization protocol as presented in Section 5 was used.
Twelve agents are split into three groups of four as shown in Figure 27. Each group is simulated on one computer. One of them uses an Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz with six cores, two other computers have an Intel(R) i7-7567U CPU. All three computers use 16GB of RAM. The machines were connected through a router to easily demonstrate what happens when agents get disconnected from each other.
The timeline summarizing the experiment is shown in Figure 28. The agents within a group are assumed to always be able to communicate. To demonstrate the ability to handle network interruptions, we simulate disconnection and re-connection between groups. Computers used for Group 1 and Group 2 get disconnected for some period of time from the router as shown in red in Figure 28. This results in creating up to three distinct groups that elect their own merge masters (c.f. the orange color in the figure) until the connection is reestablished and a single merge master for all three groups is elected. During the course of the experiment several agents make simultaneous changes to the RDF Graph which need to be synchronized. The new revisions, which are the result of the changes, are marked by blue arrows.
The results show that the agents are quick to synchronize the RDF Graphs with each other (c.f. the unsynchronised times in Figure 28) even in the context of multiple concurrent edits or connection and disconnection behavior.
Each agent received an average of 1959 messages, out of which 1584 (approx. 81%) were status messages (see Section 5.1.6). Agents received the maximum of 105 messages/s and the average of 2.25 messages/s. The final Graph of Revisions of the experiment presented in Figure 29 shows the relation between the different revisions and their creators (numbers in boxes). The red areas show the revisions that were created during the three disconnections: A, B and C (c.f. bottom of Figure 28). The turquoise area shows an excessive number of merges that were computed during the reconnection. This happens in particular at the end of Disconnection B, when Group 0 and Group 1 reconnect. The merge master, Agent 5, follows an eager strategy for merging, and starts the process as soon as it receives revisions from Group 0. After an election occurs and Agent 0 is selected, the same problem repeats. In normal operation, the merge eagerness allows for faster propagation of knowledge between agents, but in future work, different strategies for when to merge will be investigated.

Evaluating the Maximum Revision Creation Rate
In this experiment we are interested in the scalability of the RDF Graph synchronization mechanism when many agents make concurrent changes to several documents. The evaluation presented in this section is focused on finding the limits on how fast the changes created concurrently by multiple agents in multiple RDF Graphs can in practice be merged and synchronized without unnecessary delays. In other words, we are interested in the practical rate of changes below which the merge master can perform its task in a timely manner. Above this limit the queue of changes grows without the possibility to be merged on time, before more changes are created. There exists a wide range of factors which influence this maximum rate: the number of RDF ⊕ Documents, the number of revisions per document, the number of participating agents, communication links' properties, agent's hardware etc. For this experiment, we focus on the factors that directly affect the computational cost of merging, such as the number of documents, the amount of changes to the documents and the number of participating agents.
Agents perform changes to the RDF Graphs concurrently and locally as they carry out their mission tasks in real world environments. In a general case, the agents' KDB Managers create new revisions for each change in a document which is then sent to the merge master. In practice, it is the performance of executing the tasks of the merge master which limits the rate at which the RDF Graphs can be merged and synchronized in a timely manner. This issue is related to the Never Synchronized Problem described in Section 5.1.5 from which one can conclude that the rate of publishing of local revisions needs to be smaller than the time it takes to perform the merge by the master. Also note that the fact that rebase is used does not solve this practical bottleneck. It only handles the behaviour of an agent regarding local revisions while waiting for the result of a merge.
Effectively, if agents create changes to shared documents above the maximum rate at which a merge master can perform its tasks in a timely manner, a delay will be introduced in the synchronization process. The delay duration will vary depending on the amount of excessive changes that still need to be combined. In an extreme case, when all agents always create changes over the maximum rate, the delay will keep growing and the full synchronization will never be achieved. However, in a more practical setting agents join and leave missions as they have limits on how long they can operate (e.g. fuel, battery power), and the maximum load imposed on the merge master will vary over time. After a period of too frequent updates, when the rate becomes lower and manageable by the master, the changes will eventually be merged, thus the information will be synchronized among all agents. One way to improve maximum synchronization rate would be to include the computational performance of the agents when electing a new merge master, but this will be pursued in future work.
With the above in mind, the evaluation was performed using the following procedure loop: • Each agent creates a new revision with at most C insertions or deletions of triples for each document.
• Each agent broadcasts its revisions to others and waits for the master to finish the merge.
• The master performs the merge procedure. To isolate and evaluate the performance of the tasks of the merge master, in this experiment the agents are not allowed to make local changes until the merge master finishes the merge. When the merge master finishes the synchronization process and sends the new revision, the cycle may then start again. In the experiment all documents are merged by a single master. This is a worst-case scenario as, in general, separate documents can have different merge masters thus the merge performance can be improved.
Using the procedure described above the influence of the following parameters was evaluated: the number of agents N = {2, 5, 10, 20}, each making C = {10, 20, 30, 40, 50} changes resulting in N revisions to each of M = {1, . . . , 20} RDF ⊕ Documents. The procedure loop described above is performed 30 times, on a computer equipped with a six core Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz. The influence of the communication links' properties is excluded in this experiment in the name of reproducability between single merges in the loop. Average time for 30 iterations is recorded and reported in Figures 30 and 31.
The average time to perform a merge for 2, 5, 10, and 20 agents is presented in Figure 30. For each case the plots show the time as a function of the number of documents and changes per revision. For example, for N = 2, M = 20, C = 40, during the merge period, local changes can not occur faster than every two seconds, as the merge master would not be able to keep up. Figure 31 presents the same results but allows to evaluate how the system scales with the number of agents.
We believe that the performance shown is sufficient in typical use cases as agents usually generate smaller changes at much lower rates. The merge is sporadic in a typical mission because there is a longer period of time between creating new revisions. For example, in the case study presented in Section 7.

Discussion of Results
To provide additional examples and put the results of the experiment presented above in an extended and practical context, we discuss three possible use cases.
Use case 1: dataset discovery. If we consider our dataset system (Section 4), a typical scan mission takes between 5 to 20 minutes, and agents would create two revisions of the RDF Graph describing the metadata of the dataset. Assuming the mission involves N agents, 2N revisions would be required. First revision would be created at the start of the scanning mission and the second one at the end, once the dataset is complete. In this scenario, agents work with a single RDF Graph.
The results in Figure 30 show that for 20 agents, for a single document, if all agents make a change simultaneously, the merge takes 2.9s, meaning that agents can make up to 0.3 changes per second on average. In this scenario, each agent makes at most 2 new revisions per 5 minutes, which is well below what the system can handle. Extrapolating these results using the fact that the merge and rebase algorithms scale quadratically in the worst-case, the HFKN framework should be able to handle missions of this type with up to 143 participating agents.
Use case 2: victim detection. In this scenario, we consider a fleet of UAVs exploring an operational environment and searching for victims of a natural or human-made disaster. Each UAV adds the location and selected information about the identified victims in a single document. Each victim is represented by two triples, one for the victim's location and one for the state (injured, unconscious, hungry, etc.).
Let's consider the mission is performed in one of the most dense cities in the world is Manila 9 with 41515 person per square kilometer (i.e. 0.04 person per square meter). One can assume that the UAVs are flying at an altitude of 13.7m and are equipped with cameras using a wide-angle lens with 60 deg field of view. Each image acquired by a single UAV covers an area of approximately 15.8 × 15.8 ≈ 250 2 . Assuming each UAV flies at 15.8m/s speed, it can scan 250m 2 /s. Ideally, this would result in detecting 250 × 0.04 = 10people/s in the city of Manila. Consequently, each UAV would generate 2 × 10 = 20 RDF T riples/s. Instead of creating a new revision per one detected victim, we buffer the information until we detect 5 victims which results in creating 10/5 = 2 revisions per second.
According to the results presented in Figure 30, this type of mission would be feasible when using up to 20 agents working on the same RDF Graph. During a rescue operation, it is critical to conclude the exploration as quickly as possible. Given example of rescue mission in Manila performed using a small fleet of 20 agents, would result in rather long exploration time of 2h20m (Manila has a surface of 42.88km, and each agent covers 250m 2 /s). Since 20 agents is the limit, in this scenario, for collaboration on a single document, to use more agents a divide and conquer approach is used, and several documents covering smaller areas would be created.
Use case 3: exploration with a fleet of UAVs. In this scenario, we consider N UAVs scanning an area with color and thermal cameras, both delivering images at a rate of 10Hz. Additionally, locations of the platforms are recorded at 30Hz. The UAVs collaborate on three documents: one for the color camera, one for the thermal camera, and one for the location data (M = 3).
Each image is represented by three RDF Triples: one to specify which agent acquired the image, one to specify the acquisition timestamp, and one to point to the location of image data. Since images are delivered at a rate of 10Hz, and there are three triples used, agents will create 30 changes per second for each of the two camera sensors. Therefore the total number of changes to the documents about the camera data is 60 per second.
The location information is represented by three RDF Triples: one to specify whose agent location it relates to, the timestamp, and the actual position value. In that case, agents will create 90 changes per second to represent the location information. The total number of changes the agents will create per second is 60 + 90, that is 150 changes per second in the three documents (C = 150).
The agents collaborate on three documents, and they report images as well as locations twice per second which results in revisions with 30 and 45 changes, respectively.
According to Figure 30, this would work up to 10 agents. With 20 agents, it would take more than two seconds to merge the changes created during one second. As before this limitation can be overcome using a divide and conquer approach.

Summary
In this section, we have presented an empirical evaluation of the RDF Graph Synchronization mechanism. We have confirmed that merge and rebase algorithms have a quadratic complexity (see Section 7.1 and Section 7.2). We have also demonstrated the behavior of the system when agents get disconnected from each other (in Section 7.3). The experiment shows that when disconnections happen, agents can still be synchronized with a sub-group of agents and that when agents reconnect, they can handle the synchronization. Finally, we have demonstrated in Section 7.4 that the HFKN Framework scales well when the number of agents and documents increases.
The results confirm that the RDF Graph Synchronization mechanism is suitable for sharing low bandwidth semantic knowledge among agents. The results of Section 7.4 also indicate that the framework could be used with higher bandwidth data, such as storing the results of victim detection. Future work will include evaluations of different applications.
All the experiments presented in this section are synthetic and show the theoretical performance. In the following section, we present a field experiment of the HFKN Framework deployed onboard actual robots.

Field Robotics Case Study and Experiment
The proposed HFKN Framework has been implemented in prototype and validated through a series of experiments both in simulation (Section 7) and in real mission scenarios using multiple UAV platforms. The latter is the topic of this section. We first provide a short description of the UAV platforms used in the field study and then proceed to a description of a multi-agent data collection mission that uses these UAV platforms. We then conclude with a description of the rich set of query mechanisms that are part of the HFKN Framework and have also been used in the field study.

UAV Platforms used in the Field Test
Two types of UAV platforms were used for experimentation in the case study. The first, shown on the left of Figure 32, is a modified DJI Matrice 100. It has a maximum takeoff weight of 3.6kg and 1.2kg of payload capacity. The platform measures 100cm between propeller tips. It can fly with speeds up to 22m/s and has a maximum flight endurance of 22 minutes. The platform is equipped with a Hokuyo UTM-30LX LIDAR, which is a single scan device with a guaranteed range of 30m (60m maximum).
The second type of platform, shown on the right of Figure 32, is a modified DJI Matrice 600 Pro. It has a 15.1kg maximum takeoff weight, 6kg of payload capacity, maximum flight speed of 18m/s, and 35 minutes of flight time using 5.5kg of payload. It measures 167cm between propeller tips. The GPS system on-board uses a Real-Time Kinematic (RTK) positioning technique to deliver centimeter accuracy measurements. The platform is equipped with a Velodyne PUCK LIDAR sensor, which has an effective range of 100m and uses 16 scan channels. A LIDAR mounting mechanism developed and deployed on the DJI Matrice 600 Pro allows for choosing the sensor orientation depending on the applications or missions at hand.
Both platforms are equipped with the same type of onboard computer system. It is an Intel NUC Kaby Lake i7-7567U CPU platform in a custom enclosure equipped with 16GB of RAM and 500GB SSD of storage. The systems interface with the platforms and run the software modules associated with the Delegation Module and the SCModule. The communication with the ground station for both platforms is realised using 5GHz WiFi connections.

Field-Robotics Case Study: Collaborative 3D Modeling
In this field experiment, we want to demonstrate how multiple agents can explore different parts of an operational environment, share the resulting collected information, and reuse existing information from other missions. The mission leverages functionalities from the HFKN Framework and the Delegation Framework and each robotic agent has a Delegation Module and a SCModule. For human agents, it is assumed that they can interface to the collaborative system through Command and Control centers, or if in the field, through laptops, smartphones, or tablet systems with access to both the Delegation Module and SCModule. • At the end of mission C, the ground operator OP requests the transfer and download of the resulting scanned data from U AV 2 . This is done using the Dataset Transfer Protocol.
• After viewing a 3D model generated for region C, the ground operator determines that more information is required for a larger region that overlaps with the three previously specified regions, A, B and C.
• The ground operator OP sets up the fourth mission using the Delegation Framework to gather a full scan of region D by U AV 0 . Since raw data for regions A, B and C already exists, the fourth mission by U AV 0 is automatically tasked to gather data from the region For each Scan mission attempted, the following steps are applied by an agent: • First, check if any relevant scanned data is already available in the agent's KDB. This is done by executing a SPARQL query on the metadata for the dataset, checking for any intersections between the raw data areas previously covered by the agent and the target dataset.
• If no such data (or only partial data) is available in the requesting agent's KDB, an attempt is made to set up a task to copy any relevant data for the geographic region from any remote agents on the team. The copy is done through a combination of queries and use of the Dataset Transfer Protocol described previously.
• After doing this, a check is made to see if data for the geographic region of interest is complete with full coverage. If not, the agent can query its KDB to compute those remaining regions in the geographic area not yet covered. The agent can now specify a Scan mission to gather the remaining data.
Dataset content after mission's A, B and C are completed. • Before executing the Scan mission, the agent creates a dataset for the geographic area of the mission. During the execution, the agent collects data and associates it with the dataset.
• At the end of the mission, the agent would have the necessary raw data in its KDB for the geographic region in question. Additionally, due to the periodic synchronization of knowledge in each of the team agent's KDB, they would all have a new dataset with information about where the raw data resides in other agents.
Before mission A, B and C, no data for these regions is stored, either by the individual agents or the team as a whole, so there is no exchange of raw data. Each UAV collects the requisite data associated with its respective mission during the execution of each mission and stores it in its local KDB. We will refer to the resulting datasets as D A , D B and D C . No raw data is exchanged between U AV 0 , U AV 1 , U AV 2 at the end of their missions. Metadata for D A , D B , and D C has been synchronized in the KDBs of all team members.
At the end of mission C, by request of operator OP , raw data for the dataset D C is copied to the operator OP 's ground station using the data exchange protocol. The association between available datasets before the start of mission D is shown in Table 1a For the mission D, the OP operator queries its datasets' metadata and finds that datasets D A , D B and D C overlap with the operator's area of interest. Since OP already has a copy of D C , it only needs to download raw data from D A and D B and set up a mission for the remaining region D D \ (D A ∪ D B ∪ D C ).
The TST generated for mission D is shown in Figure 34. First, the ground operator's station should download concurrently datasets D A and D B . Then, it delegates the final UAV Scan mission to collect the remaining scan data to U AV 0 . This TST is the actual TST generated by the delegation framework and intended to be executed by the ground station agent and U AV 0 autonomously (The S and C in the control nodes of the TST refer to Sequential nodes and Concurrent nodes, respectively). To demonstrate our system's robustness, we simulated a communication failure between agent U AV 1 and OP , so that D B was not transferred to OP during mission execution. OP is made aware of this. It would either need to attempt a new data exchange mission or start a new exploration mission to make sure that D D covers the requested area. Table 1b shows the raw dataset associations to team agents after mission D is completed. U AV 0 has performed two scan missions: for region A and for the latest request from the operator. That is why it has a copy of the raw data associated with datasets D A and D D . At the end of mission D, all metadata associated with all datasets generated is synchronized, and general queries about the operational region can be made by the agents of the team, where each has a partially shared situation awareness by virtue of each agent's sharing respective metadata about datasets. Figure 35 shows the resulting point clouds for each mission as well as the final point cloud accessible to the ground operator OP . Here one can observe that point cloud data for region B is missing. This was due to the communication breakdown.

Querying and Query Capability
The HFKN Framework includes powerful query capability for human and robotic agents to acquire information about operational environments in different modalities. The resulting answers to queries can be Elementary tasks to be executed used not only for situation awareness and decision making, but also to specify information gathering missions based on missing information. This was shown in the case study. We now consider query mechanisms [89] in more detail and provide some additional examples. Robotic or human agents, can query one agent individually using SPARQL, or several agents simultaneously using Federated Queries [73]. From a mission perspective, a ground operator's queries play a special role where multi-modal interfaces can be used to advantage. In this case, rather than the human agent specifying SPARQL queries in detail, a graphical user interface has been developed that permits human agents to define queries graphically. For instance, given a map of an operational environment, a human operator can draw regions of interest (ROI) graphically, state what kind of information from that area they would like and press a query button. The interface then translates the graphical query into a SPARQL query which is directed to one or more agents on the team. The query reply can then be depicted both graphically and textually on the operator's screen.
The execution of a SPARQL Query from a ground operator perspective uses the following components (shown in Figure 36): • User Interface -A user selects an area of interest and a data type of interest (e.g. list of buildings with their 3D models, potential victim locations with pictures, etc.). Based on that selection the User Interface then automatically generates an appropriate SPARQL Query.
• SPARQL Engine -The SPARQL Engine converts the SPARQL Query into a corresponding SQL Query. After the query is executed it also converts the result back into a SPARQL Result.
• SQL Engine (in this case PostgreSQL) -The SQL Engine is responsible for execution of the SQL Query over the selected RDF Graphs (either stored as SQL tables or accessible through an SQL View).
• RDF View Manager -The RDF View manager then generates an SQL View based on an RDF View definition, i.e. mapping any SQL Tables to an RDF Graph representation.
When a user selects an area of interest in the User Interface, together with a data type of interest, a SPARQL Query is generated and sent to the SPARQL Engine which converts it into a SQL Query over a virtual table with three fields (subject, predicate, object). This virtual table is the union of a set of RDF Graphs (represented as SQL Tables) and a set of RDF Views (mapping between any SQL Tables and an  In the case study, we showed how to collect information through specification of geographic regions or boundaries. This idea can be leveraged in many different ways. For example, suppose an operator or human rescuer is interested in knowledge about surrounding building structures and would like 3D models of existing building structures. The following query example shows how this can be done using the data from the case study missions. In the case study, LIDAR sensor data and intermediate point cloud data were collected by several UAV agents and stored distributively in their respective SCModules. Additionally, a human ground operator acquired some of this data through SPARQL queries and the use of the data exchange protocol. During this process, raw sensor data was stored in the agent's respective PostgreSQL databases in table form (see Appendix A). Using the RDF View system, this data can be accessed using SPARQL as a high-level query language.
In the following, the prefixes before the URLs will be used for defining the RDF View and SPARQL query used to retrieve 3D models of building structures in a geographical region: • askcore pointclouds: http://askco.re/pointclouds#. This prefix is used for definitions related to point clouds. askcore pointclouds:patch is the predicate that specifies the raw data of the point. askcore pointclouds:intersection is a function that computes the interesection of a point cloud with a geometric object. askcore pointclouds:union computes the union of a set of point clouds.
• askcore graphs: http://askco.re/graphs#. This prefix designates the RDF Graphs used in our system. askcore graphs:pointclouds view is an RDF View which enables one to query point clouds stored in SQL Tables using SPARQL queries.
An RDF View mapping the SQL table for point clouds to an RDF Triple representation using Sparqlify [78]

Related work
The HFKN Framework envisioned is unique in its approach, but does take inspiration from some of the challenges and problems in related topic areas. In this section, we consider some of this work most relevant to the HFKN Framework.
Distributed Knowledge and Multi-Agent Systems. Multi-Agent Systems (MAS) [88,76] have direct relevance to the HFKN Framework since we view each robotic system as an agent with capabilities to plan and reason. In MAS, the topics of distributed and common knowledge are central to logical approaches to multi-agent problem solving and inference [41]. In the HFKN Framework, each agent has its own local knowledge in the form of RDF Documents/Graphs that are distributed across agents. The multi-agent system consisting of a team of agents has common knowledge in the form of the shared, synchronized RDF Documents/Graphs. Collections of RDF triples (RDF Documents) can be viewed logically [46,17,18]. This being the case, certain kinds of querying mechanisms, analogous to deductive databases [45], can be viewed as doing logical inference. SPARQL-DL [77] is an interesting example of this.
Communication among agents is also an important topic in MAS. A common approach for communication in MAS is to use an implementation of the Agent Communication Language (ACL) such as FIPA ACL [66]. ACL contains mechanisms for agents to query other agents and transfer beliefs between them. This provides an infrastructural mechanism for belief transfer but also requires a semantics for beliefs such as that used in the BDI approach [15,70]. It should be noted that the Delegation Framework and its integration with the HFKN Framework, does use ACL and speech acts for general communication between agents and that general communication mechanism carries over for use with SCModules.
One of the challenges with such multi-agent systems that is related to belief transfer is for agents to determine what agents to ask for certain information required during a mission. Sending a broadcast is not always practical, since for large numbers of agents, this approach could degrade the available bandwidth of the communication network. A potential solution involves using matchmaker agents [21] that hold information about which agents can answer questions. Our approach is much different because each agent's public/shared information is synchronized, so the collective knowledge from a team is accessible directly to each agent. Consequently, each agent can then internally query such information. Raw sensor data is indirectly accessible through metadata and dataset transfer protocols, but an agent can still directly query a dataset's metadata to acquire useful knowledge.
Distributed and Federated Database Systems. The main challenge in Federated Database Systems (FDS) [75,8] is to provide a unified query mechanism that hides data inconsistencies. This is often called the database integration problem [64]. The main challenges in Distributed Database systems [63,61,24] is the consistency problem where schema definitions and data should be consistent and equal across the different database instances.
Traditional database technology provides common techniques for storing data and querying data. For databases running on a single computer and using smaller amounts of data, data consistency is less of an issue. But this does not scale for large numbers of users and large amounts of data. This has led to the development of Distributed Database Management Systems (DDBMS) [63]. Homogeneous DDBMS are systems where the schema definitions and data should be consistent and equal across the different database instances. A common approach to solving this consistency problem is to use database replication [61], where a master has write permissions on a subset of the data, and when changes occur, they are propagated to the slaves. Homogeneous DDBMS also solve the problem of load balancing between servers. In the case of Big Data, when a single computer cannot store all of its data in memory, it is necessary to use a Heterogeneous DDBMS approach, such as Spanner [24] or Dynamo [30]. Here, the system controls which server stores which data depending on user needs and system requirements. In [24], the authors propose a dynamic system to lock tables so that any instance can write on a subset of the table schema. This approach is also capable of handling inaccuracies in the timestamping of transactions. In [30] the data is replicated across multiple hosts. Dynamo trades-off consistency for availability, where data is guaranteed to be eventually consistent, that is all updates reach all replicas eventually. Generally, these systems rely on a central server or policy for handling the spread and distribution of data. Their goal is to optimize the efficient accessibility of data by end-users and deal with load balancing between servers.
In Homogeneous and Heterogeneous DDBMS, the control on the availability and writing of data is left to the system. In Federated Database Systems (FDS), each database instance is autonomous, in the sense that there is no assumption as to the individual schema and the availability and location of where specific data resides. What is of particular focus is the development of a common query mechanism across heterogeneous databases. Such systems have to deal with many types of data inconsistencies, such as naming of concepts, precision, schema alignment, etc.
FDS attempts to provide unified query mechanisms that hide data inconsistencies in order to deal with the database integration problem [64]. A solution to the distributed mapping of integrated answers to queries was proposed in [75], where the system is required to be static, and each database instance must remain in the federation. A more dynamic approach was recently proposed in [8], where the FDS is implemented as a graph. When a query is executed, it is propagated along the graph and at each node the results are aggregated, correcting the inconsistencies incrementally.
The HFKN Framework shares some ideas from both DDMS and FDS technologies. The SQL database associated with each agent is not intended to be shared, globally consistent or replicated across agents. These are heterogeneous databases by design. On the other hand, the schema for representing sensor data of any kind is common to all agents in the system. Even at the RDF document/graph level intended to maintain semantic representations and meta-information about datasets describing collected sensor data, the HFKN framework does not require complete consistency. A form of weak consistency is guaranteed as discussed in Section 5.5.
With FDS, the schema (i.e. structure of the data) and concepts (i.e. the meaning of the data) are the same for all agents. In the HFKN framework, each agent has full autonomous control of what kind of data it stores. Agents can join and leave the federation at any time. This is different from previous work [75] with FDS where an agent can leave a federation only after obtaining a permission. In our application scenarios, communication is assumed to be unreliable. Consequently, the approach proposed in this paper is designed to handle dynamic changes in the federation structure without any notice.
Like FDS, our approach does use a unified query mechanism, SPARQL (SPARQL Protocol and RDF Query Language) [74] for this collection of shared RDF documents/graphs, but also locally for internal agent queries of all its RDF documents, shared and unshared. SPARQL was developed for querying information stored in RDF Graphs.
Semantic Web and Robotics. Our choice of representation for information and knowledge, RDF documents/graphs, is inspired by Tim Berners-Lee's vision of a Semantic Web [10,47,2], where web pages in the World Wide Web are annotated with representations of their semantic content in the form of collections of RDF triples (RDF documents) and ontologies and linked across the WWW. These information structures can be reasoned about using logical inference mechanisms such as those based on Description Logics [3], in addition to related powerful ontology inference mechanisms such as OWL [59,3]. The modern equivalent of these ideas has resulted in standardization [1] and the linked-data [87] research area of which knowledge graphs are a prime example. Many additional tools and technologies have been developed since the original idea of the Semantic Web was proposed and these are used in the backbone of many companies' knowledge-intensive products. More recently, there has been a trend toward leveraging Semantic Web ideas with robotics [55,79,62,54,65,56]. Many existing ontologies are available and useful for describing robotic systems [72,19] and sensing for robotics [62].
Several frameworks using Semantic Web technologies have been implemented on robotics systems. The OpenRobots Ontology (ORO) [54] presents a processing framework leveraging Semantic Web technologies to enhance communication between robots and humans. ORO makes use of a common sense ontology, an events system, a model of the cognitive state of other agents, and a long term memory. In [56], the authors present a method to connect low-level sensor data and high-level semantic information using RDF and an ontology for low-level information.
The HFKN Framework relies heavily on Semantic Web technologies. This choice is due in part to the conceptual fit between our ideas and these technologies and the wider availability of collective knowledge on the Internet provided in the form of ontologies and RDF Documents/Graphs, the latter often called knowledge graphs. Due to the similarities, these knowledge sources can be leveraged, integrated, and used to best advantage by the collaborative agents in the scenarios we focus on.
One of the most mature frameworks combining Semantic Web technologies and robotics is KnowRob [6,80], a knowledge processing framework for robotic agents. KnowRob supports reasoning over semantic information while taking into account planning processes. KnowRob needs to be used in conjunction with a knowledge base such as the Robot Scene Graph (RSG) [13,12]. RSG is a framework for representing 3D information about a robot's environment in a graph structure stored in the volatile memory of the robot. Work that combines RSG and KnowRob is presented in [90].
In the HFKN Framework, an agent's SCModule stores information in its PostgreSQL database in the KDB. This architecture results in the information being stored on the hard drive whereas continually used information is cached in memory. The benefit of this is that the information is saved long term and is persistent even when a particular robot agent goes offline. The information is not duplicated but shared across the modules of a robot. However, there is additional overhead when accessing the information stored in the PostgreSQL database when compared to the in-memory solution used with RSG. Still, efficient caching minimizes this as an issue.
RSG has recently been extended with a synchronization mechanism in [13], which allows the distribution of RSG nodes across a team of agents. Since some of the information stored in RSG can be large (e.g. point clouds), the authors have implemented a QoS solution that checks the network's quality and then downsamples the information relative to network congestion levels.
The HFKN framework deals with much more general forms of synchronisation then those provided in RSG. Additionally, only low bandwidth information is automatically exchanged between agents through synchronization. Large datasets (e.g. point clouds and images) are only transferred after an explicit request from an agent. This results in a by-need philosophy, where agent SCModules transfer full resolution information only when needed. Note that there is a throttling mechanism that controls data transfer bandwidth when using the data exchange protocol. This is, in fact, a form of QoS mechanism. One could envision combining these ideas with the QoS techniques used in RSG, but we save such issues for future work.
Research in the area of Cloud Robotics [85,51,14] is also of interest, but less directly related to the specific challenges in this paper. Additionally, the work in both Cloud Robotics and Semantic Robots has focused primarily on single robotic systems rather than dynamic teams of robots as the HFKN framework does.
Applications with Mapping. The topic of Distributed Simultaneous Localisation and Mapping (SLAM) has some relation to our applications of the HFKN Framework. One of the basic functionalities in any robotic platform is building a map of the environment. Many algorithms have been developed for a single robot [40,4], which deal with the problem of fusing information coming from multiple sensors in a robot. Such algorithms have been extended to support fusion of sensor data from multiple robots. Earlier work involved considering a set of local maps connected in a global graph shared between robots [83]. This approach was superseded by introducing landmarks directly in global graphs [27]. However, such approaches require a central location for the fusion of the global map. A decentralized approach was proposed in [22]. The authors suggested using a version control system for the map, but this still required a lock when fusing part of the map distributed between platforms.
A practical application of building a common map by a group of robots is to solve the problem of exploration of a large environment [84]. Most of these approaches use either a centralized server or a global lock, while HFKN is a fully decentralized system. The lock is implicit in the synchronization algorithm. The approaches considered here are specific to solving the distributed fusion problem, whereas HFKN is a much more general framework.
As mentioned earlier, the KDB is GIS (Geographic information systems)-like in concept since it stores low-level sensor data, intermediate information, and high-level semantic knowledge. Although the KDB architecture is not layered in the GIS sense, SPARQL queries about specific geographical regions return information about those regions at many different conceptual layers.
GIS systems can be used to store a wide range of vector information (points, lines, polygons, etc. [26]). They have also been extended to allow storage of 3D information such as point clouds [28] or point clouds augmented with semantic knowledge [67]. The latter is directly related to how the KDB can semantically label lower-level data geographically.
As the state of an operational environment is continuously evolving, it is natural to use remote sensors to update GIS [23]. In [60], a centralized GIS server is used to share rescue information, where autonomous robots push observations to the GIS server and where rescuers can then access that information. As discussed previously, the HFKN Framework collects data and knowledge dynamically via agents yet stores it in a distributed manner. The HFKN Framework uses a weak form of replicated centralization in that each agent shares metadata about other agent's public data and knowledge. The synchronization process keeps this data and knowledge weakly consistent across agents. There is no centralized server or source of global information.

Conclusion
We have presented a general system architecture and framework (SymbiCloud HFKN Framework) for supporting dynamic, distributed situation awareness for heterogeneous teams of human/robotic agents. The system builds upon previous work using a delegation framework for dynamic task allocation for collaborative human/robotic systems that is also part of the general system architecture.
The major focus of the paper has been the description and specification of core system algorithms for supporting distributed data and knowledge collection and the storage, synchronization, aggregation and transfer of such data and knowledge among teams of collaborating agents. In this case, our teams have consisted of humans and UAVs.
Major effort has been placed on incorporating resiliency into the algorithms and the resulting engineered system to deal with difficult problems associated with unreliable communication among agents, out-of-range issues, agents entering and leaving mission environments, in addition to the heterogeneity of both the data collected and the systems used to collect such data.
A seamless unification of data, information and knowledge has been achieved through the use of Semantic Web technologies as a basis for structuring and querying collected sensor data, meta-information about such sensor data, and post-processed knowledge in the form of RDF (knowledge) graphs. The system can easily be extended with additional ontologies other than those described in the paper to provide richer semantic decision support.
Although the focus of this paper has been on the dynamic collection of sensor data for constructing 3D maps and limited object representations such as people and building structures, the system is general in nature and any type of semantic information that can be modeled using RDF (knowledge) Graphs could be collected, represented, processed and queried in a similar manner.
Empirical experimentation in simulation has been performed showing the speed and scalability of the proposed system and algorithms. Additionally, limited field experimentation has been done using emergency rescue scenarios with teams of UAVs and humans to show the viability of the ideas in the engineered HFKN system.

A. Representation of Data Structures in SCModules
In Section 8.3, Figure 36, we provided an overview of different ways to query an SCModule using SPARQL. In this appendix, additional details are provided concerning some of the basics Schemas and RDF Views used in the HFKN Framework, together with a selected set of SQL table representations.
Triples table. An RDF Graph consists of triples (subject, predicate, object) which are stored in a SQL table with three fields presented in Table A.1. The SQL table fields correspond directly to the components of a triple (see also Figure 8). The type of the object field provides a means to efficiently store various data types such as integers, strings, lists, images, point clouds, etc.

RDF Triple Type
Name Comments string subject Subject of the RDF Triple. string predicate Predicate of the RDF Triple. any object A complex type which allows to efficiently store any type of data, ranging from integers, strings, lists, point clouds, images, etc. RDF Document. An RDF Document is stored using three tables: • A triples tables corresponding to a version of the RDF Graph, which is available for answering queries or performing change operations (see Table A.1).
• A revision table containing the metadata associated with each RDF Revision (see Table A.2).
• A delta table containing the RDF deltas (see Table A.3).
The database also contains metadata about the RDF Document: • The revision hash of the current instance.
• The URI used to identify the RDF Graph.  RDF Delta. Deltas are encoded using a subset of SPARQL Update query [74]. Only INSERT DATA and DELETE DATA queries are allowed, because they are the only two types of query to unambiguously identify the inserted and removed triples. RDF Blank Nodes 11 are encoded using the skolemization process [58], i.e. blank node are replaced by an unique URI assigned by the KDB Manager.

RDF Revision
An example of an RDF Delta with one triple being added and one removed is encoded as a SPARQL Update query in the following way: 11 Blank nodes can be used to identify an element of an RDF Triple without providing an explicit URI RDF Delta Type Name Comments uint8[] parent The unique hash of the parent revision. uint8[] child The unique hash of the child revision. string delta A SPARQL Update representing the change between the parent revision and the child revision.   [43] for representing coordinate frames of all the agents. TF allows for keeping track of all transformations between different components of a robot (and between robots) represented as a tree. Each node in the tree corresponds to a frame and includes the rotation and translation between that frame and its parent (see Figure A.1). The root of the tree is usually the origin of the world.
The definition of the origin of the world is agent-specific and can be different for all agents. Therefore in SCModule, we cut the tree at the main reference frame of each agent and record the corresponding geoposes as different entries in the Position table. This allows for maintaining a global coordinate frame for all agents in the system. For example, in Figure A  Images. Image data is stored in two tables: CameraFrame and CameraInfo presented in Table A. 5. The former contains basic information about the parameters of an image such as the time of acquisition, its size, encoding type, etc. The most bandwidth-intensive part is the data (i.e. pixel values). The latter contains calibration information of the camera sensor, such as the matrix of camera intrinsic parameters K as well as the lens distortion parameters D.
Point clouds. Sensor data representing point clouds is stored in one table presented in Table A.6. Point cloud data typically consists of many measurements, and storing them as individual records in a database is inefficient. Instead, we use data representation proposed in [69]. The base type PcPoint is used to encode a  [3] translation Translation between the parent and current frame. float [4] rotation A quaternion representing the rotation between the parent and current frame. Position Type Name Comments string agentUri URI of the agent. float latitude Latitude of the agent in the WSG84 [31]. float longitude Longitude of the agent in the WSG84. float altitude Altitude of the agent using the EGM96 [86] gravity model. float [4] rotation A quaternion representing the orientation of the robot, such as x-axis, y-axis and x-axis are pointing respectively North, East and up.  single point LIDAR measurement. A collection of those points is then grouped in a complex type PcPatch which is stored as a record in the database reducing the total number of entries. The RDF View defining the mapping between the table and corresponding RDF Triples used for querying of point cloud data is shown in Figure A.2.
LIDAR data. LIDAR data is stored in two tables: LidarFrame and LidarConfig presented in Table A.7. The former contains information about the time of acquisition, a pointer to the sensor's configuration, and the actual LIDAR readings in the form of ranges and return intensities. The latter contains information about the sensor configuration used during the data collection. Most importantly, angular and range resolutions are specified. The RDF View definitions for the mapping between the SQL Tables and a set of triples that can be queried with SPARQL are shown in Figure A. distortionModel Name of distortion model. float [5] D 2 radial and 3 tangential lens distortion parameters. float [9] K Intrinsic parameters, 3x3 row-major matrix.    Query for the range data of a LIDAR sensor. The following query returns a set of ranges from the specified dataset, i.e. ?range from the dataset ex:dataset1 : Query for acquiring a list of buildings and their 3D models. This is an example of a query to access a list of buildings ?building_name and their corresponding 3D models ?building_3d_model : SELECT ?building_name, ex:filter(ex:combine(?point_cloud_3d), ?building_2d_footprint) AS ?building_3d_model WHERE { ?building_uri a ex:building ; geo:Geometry ?building_2d_footprint ; foaf:name ?building_name . ?point_cloud_uri a lrs_sensing:point_cloud ; geo:Geometry ?point_cloud_2d_footprint ; ex:data ?point_cloud_3d . FILTER(geo:intersects(?building_2d_footprint, ?point_cloud_2d_footprint)) } GROUP BY ?building_name The first part of the query lists all buildings. Their 2D footprints are stored in ?building_2d_footprint and their names in ?building_name : ?building_uri a ex:building ; geo:Geometry ?building_2d_footprint ; foaf:name ?building_name . The second part of the query lists all the point clouds ?point_cloud_3d whose footprints ?point_cloud_2d_footprint were obtained by projecting all the points on a 2D plane intersecting with a specified building footprint ?building_2d_footprint : ?point_cloud_uri a lrs_sensing:point_cloud ; geo:Geometry ?point_cloud_2d_footprint ; ex:data ?point_cloud_3d . FILTER(geo:intersects(?building_2d_footprint, ?point_cloud_2d_footprint)) Finally, the point clouds are combined to give a single result for each building with: ex:filter(ex:combine(?point_cloud_3d), ?building_2d_footprint) AS ?building_3d_model The ex:combine function takes a set of 3D point clouds and combines them into a single one. The ex:filter function takes a point cloud and a 2D footprint and output a point cloud for which each point is inside the footprint.