Keywords

1 Introduction

Emergency events require building effective operational responses. Frequently, unexpected incidents arise and demand time-critical decisions from a specialist of the state police, security managers or governmental members. Such decisions involve the deployment of new tactics and the allocation of human resources and equipments.

The assessment systems of emergency events are highly complex due the need of comprehension by specialists about what is going on at the event location. Such comprehension is supported by Data Fusion systems, feed by multiple sources (hard sensors, social networks, databases, etc.) offer a more precise notion but aware about what is happening in an environment [1].

Devices and innovative fusion algorithms are being used for better supporting the assessment process for situation awareness for decision-making. Such assessment systems may have to be used even under informational adverse conditions of uncertainty, demanding that non-explicit information must be inferred. The specialist of an assessment system has a crucial role on the improvement of information that are processed from heterogeneous sources and data fusion engines for the acquisition of situation awareness. Data mining, classification algorithms and information quality are aspects of extreme relevance in this process and present several challenges regarding the determination of synergic information and the definition of fusion criteria [2, 3] and consequently new relations among information.

In the literature, different solutions have been succeeded pro- posed and implemented, reaching information with several data and information quality issues. As a result of recent technological developments e.g., proliferation of diverse algorithms and platforms, such as Nearest Neighbors, Probabilistic Data Association, The Kalman Filter, Semantic Methods extends systems abilities to develop a more precise, complete, consistent and timely information in a minor dimensionality. In a wider view, information fusion aims to provide a richer knowledge to promote the acquisition, maintenance and resumption of situation awareness of specialists in a variety of contexts [4].

However, as far as the authors investigated, there are few studies that deal more specifically with semantic information as input and the construction of an incremental knowledge for the identification of new entities and relations among them.

Intelligent systems to support specialists in critical situations (that could compromise lives and patrimony) can benefit from the diversity of available criteria (including quality) for exhaustive fusion to improve the evaluation of critical situations, reducing imperfections of information and present a reliable selection of integrated possibilities.

To overcome some of these challenges, the contributions of this paper are: (1) An intense relation with a Knowledge Representation phase, by managing knowledge as input and output; (2) The use of customisable multi-criteria parameterization, including information quality indexes and Semantic information (relations among entities) to reveal new objects, attributes and relations among them.

This paper presents a Multi-criteria Fusion of Heterogeneous Data for Improving Situation Awareness on Emergency Management Systems. It is also presented the relevance of information quality issues in the process when imperfect data can be a potential impact factor on the quality of the decision-making.

The paper is organized as follows: Sect. 2 presents the Research Background on Situation Awareness; Sect. 3 describes the applicability of Data Fusion on the Emergency Domain and Sect. 4 presents the Multi-criteria Fusion of Heterogeneous Information, followed by a Case Study and Conclusions.

2 Research Background

Situation awareness systems, specially applications Emergency Management Systems rely on information quality to provide specialists a better view of the analyzed scenario for making quality decisions. If imperfect information is provided to SAW systems, the specialist may be uncertain on what he perceives and understands and the quality of the decision will be compromised. For such, this paper presents a framework for Improving Situation Awareness comprising of a multi-criteria fusion of heterogeneous information to mitigate uncertain information about situations and assets and to use and represent enriched knowledge [5]. Hence, this combination will provide a comprehensive fusion framework for coupling into situation assessment systems for specialists deal with adverse conditions of information quality.

2.1 Situation Awareness in Emergency Management Systems

In the C2 domain, there are some elements of awareness that individual specialists must posses, such as mission goal awareness (knowing the current state of a mission and the goals current being attempted); system awareness (knowing about how the system works and its methods of operation); resource awareness (knowing the available physical and human resources) and in case of workgroup, team awareness (knowing that all team members know the state of current events) [6].

The way in which information is combined by the system and also by the specialist through the interface and embedded visualizations influences human’s SAW and consequently the quality of the decision. A suitable SA-oriented fusion should determine which information can be synergically integrated in the event timeline, perform an accurately search for the input information and also be committed to deliver an information that is conformable with the specialist’s SAW needs and expectations [7].

Data fusion systems are able to deliver the needed processing to provide information for SAW purposes, however, they still require the human participation on the processes for the interpretation of the results produced by such systems and give meaning and relevance for information [8]. Supported by a user refinement phase present on situation assessment models, opportunities referring to the customisation of fusion criteria became imminent given the need for knowledge about data and produced information, strengthening possibilities of effective contributions to the data fusion process [6, 8, 9].

2.2 Information Uncertainty

Applications of data and information fusion systems are then able to process and present data with the objective of enabling specialists to make effective decisions. However, such systems can propagate imperfection due to fail data acquisition, processing or even in the representation of the produced information, making user uncertain about what s/he perceives and understands.

To overcome such issue, innovative fusion techniques must be present in the assessment system to provide a truly knowledge representation about situations. Also, quality issues must be always known in order to build a better SAW [10]. In a decision making complex environment, commanders need a clear, concise and accurate assessment of the current situation, what may be degraded by an uncertainty about where the assets are located, what are their capabilities, the nature of their intentions and if there is any kind of risk to people or patrimony.

Often, the specialist does not have confidence on his knowledge about his own forces and much less about the enemies, mostly due information unavailability, the dynamics of real scenarios with ever changing situations and the distributed nature of data sources [6, 11]. In summary, when monitoring assets in a conflict scenario, specialists have to pay attention to all assets parameters they had monitored in the past, how long the assets must be monitored, which assets parameters must be monitored in the present, whether new information about the assets are certain and reliable, how to combine new information with the existing information that the specialist already possesses, what parameters must be shared and how to use new information to make future decisions.

3 Information Fusion and SAW in Emergency Field

SAW is the ability of understanding and project a real monitored environment, in this case by an operator of the emergency system, who based on the SAW developed during the observation of the support system, will make a decision.

According to Endsley, SAW is being aware of what is happening around you and understanding what that information means to you now and in the future [2]. In the emergency field, on a particular situation, SAW is developed by means of an enormous range of information collected from various sources, and for the most part, in real time and presenting quality problems in their composition, thus affecting their representation to the operator. With this huge amount of information to be observed by the operator, the development of SAW is difficult and weak, since it still presents quality problems, and it does not have a system and appropriate interfaces to properly represent all that data.

SAW composition presents different aspects for every need or environment, since the perception of information is made by visual, hearing, and tactile stimuli, among others, and also a by a combination. However, in order to supply and feed a SAW system, especially in a critical and complex system such as emergency decision-making, these systems require quality information in several stages that make up a process for evaluating situations, in several dimensions that determine the scope of application.

The role of information fusion routines is to apply techniques that contribute to mitigate such quality problems. It is known, however, that many of the problems depend on the activity of the human expert, and that its input is extremely valuable for the process. In this manner, the expert should be free and able to propose new fusion processes not provided by the system, with new parameters and attributes, and between objects often considered completely different, but that can still account for a lot of valuable information to process SAW and the decision-making process.

Thus, the present paper proposes a multi-criteria and customizable fusion process, able to merge and link data before tried, with no association or irrelevance to the situation, managed by the system and the expert.

4 Multi-criteria Fusion of Heterogeneous Information

In order to fulfill the objectives proposed in this paper, we propose a framework of activities that organize the process of merging information aiming the SAW for a decision-making system of São Paulo State Police (PMESP), and incidences management, and checking of criminal situations.

This framework can be abstracted into four main processes, each of them with internal mechanisms that play specific roles, but of great contribution to improve the quality of information, and therefore the SAW. The main processes (highlighted in Orange in Fig. 1) are, Acquisition (Sect. 4.1) and the Information Fusion (Sect. 4.2), described in detail below.

Fig. 1.
figure 1

Framework for multi-criteria fusion of heterogeneous information

Full size image

4.1 Acquisition of Audio and Pre-processing

In the proposed framework, there are several data sources, such as: audio of a complaint call to the police, social networking posts, databases, and images. Each scope has particularities of the sources adopted for the completion of the evaluation of a situation. In our case study, we will debate how audiotapes, posts, and security camera images can be implemented.

When using audio as input data to the framework, data acquisition is made using the Speech to Text tool. The audio is then captured and transcribed at runtime to a string object. Every word captured is sent to Google’s servers that return the spoken words.

At the end of speech, the string containing the transcribed words is sent to the server, which prepares it and insert it into the base, creating a structured information. The string is sent along with the identification metadata of the complaint to a Java server.

A Java server co-running with the PHP server receives the complaint containing the expected identification information and the string containing the transcribed speech. When receiving the data, the string is analyzed using the CoGrOO API, which is a grammar checker developed in Java.

Using the CoGrOO source code alone, it is possible to add Tags such as nouns, number, object, and every other word. We can also join the phrases obtained in the input text. After the classification of words and phrases association, it is made a post to the PHP server using a JSON (JavaScript Object Notation) object containing the analyzed metadata phrase and the classifications made.

In this way, the server can recover the data saved and begins the analysis in search of the elements already defined as important when checking the requirements. By analyzing the classification of a word, it is possible to infer that what comes next may be a class object such as addresses, names, etc. To determine the next possibilities of a word, it was analyzed several words, and a glossary was gathered in order to meet the needs of the occurrences. By using keywords, it is made the linking and the classification of each object found and their attributes, for instance, place, address, number, type.

All data are re-factored within the system, thus suggesting changes based on the several words found during the analysis, resulting in increased accuracy and quality with the use of the system.

At this stage, the identified objects are encapsulated in a JSON object, shown in Fig. 2, and subject to the process of quality assessment of information, and later to the merger information.

Fig. 2.
figure 2

JSON after process acquisition

Full size image

The objects, attributes and properties generated by this stage are then submitted to an evaluation in order to quantify the information according to completeness, timeliness and uncertainty dimension, better discussed by Souza et al. [12]. At this stage quality scores are assigned to information aiming to inform the operator about such measures. Such scores are also used as Fusion criteria to be discussed in the next section.

4.2 Information Fusion from Acquisition

After the acquisition and quality assessment, an object in JSON format (JavaScript Object Notation) is produced. This object corresponds to objects and attributes level L1 of fusion (according to the JDL taxonomy), along with quality scores assigned to each object and attribute, according to the methodology of Souza et al. [12] for quality scores of objects, attributes and situations identified in acquisition time.

From the JSON object decoded, it is instantiated a preliminary ontology with the classes of victim, criminal, robbed object, and place, considering the fundamental classes to define a situation, each with their respective attributes and relationship properties, representing the semantic meaning of the information, as shown in Fig. 3:

Fig. 3.
figure 3

Ontology of situation

Full size image

With the instantiated ontology, information set within such ontology is once more transformed into a JSON object, now with the semantic meaning between objects and attributes. These semantic information are related by means of the present properties.

This JSON will be the input of the fusion. In addition, there will be other input parameters, and criteria set by the operator such as: which objects to specifically associate, which external sources to use, which properties must exist, and a new threshold of minimum quality of the information, as shown in Fig. 4:

Fig. 4.
figure 4

Input multi-criteria by operator

Full size image

Once the fusion process is started, there is the search for synergy information among classes, which are already present in the ontology, complaints, posts, and camera metadata, taking into account the objects, attributes, quality scores, and their properties set into the ontology.

After the search for synergy information among the information already stored in the ontology, a search of other sources of data will be carried out in order to obtain new information about the associated objects in each moment, and validate and give greater consistency to the already set information. Considering that the input of this process is isolated information or information already associated among objects (level L2), the result of this process will be a L2 level fusion.

This process will be carried out by an algorithm based on data mining techniques, based on the algorithm a priori [13], which infers the frequency of certain information when analyzed in relation to the rest of an environment (information of the situation and requested other sources), this inference is made from a proposed calculation of the information support, according to Fig. 5, as follows:

Fig. 5.
figure 5

Part of the core of data mining algorithm

Full size image

For the process herein proposed, it will be used the principles of data mining, but some changes will be made in order to analyze the frequency of such information by a single knot or proposed parameter, that is, it will search for the covariance between the data and will calculate this relationship index. It is the calculation of the total of all items related, divided by the total of items found in all the information.

The result of this process is obtained by the initial information plus the new information with their respective attributes, quality scores, properties, and the relationship score among such information.

This result is validated in the next step (multi-criteria association), which deals with the association between the information submitted to the fusion process and the synergy information found. This association meets certain pre-defined criteria such as quality scores and properties pre-defined as satisfactory for each association or type of information. The following process inserts the new information found within the context of the original information submitted to the process, in order to achieve and meet the multi-criteria defined in the previous process. Figure 6 shows part of the core of the multi-criteria association algorithm.

Fig. 6.
figure 6

Part of the core multi-criteria association algorithm

Full size image

Multi-criteria result from two sources as an input of the expert agent entry in times of system operation, or they are based on requirements analysis conducted during the project.

As a result, every initial information submitted to the fusion process is generated, however, with new attributes, properties and even new objects found during the fusion process, thus making results explicitly L2 level (Situation Assessment).

This result can be submitted again to the previous process of search for synergy data, then increasing its capability to find new information and consolidate the information already available. In this manner, information increasingly specializes in the actual current situation, or in a single part of the situation, depending on the input. This cycle will be performed when the result of the multi-criteria Association, objects, attributes, indexes of quality and properties, are not within the parameters required, defined by human input or delimited by the system.

If this second process is not carried out, the resulting information will be submitted to the quality assessment process, now scoring the new information found and reconsidering the general scores of information. After this process, the information re-evaluated by the quality assessment will be re-instantiated in the ontology and represented in the system, according to the request of the operator.

In the primary fusion phase, in the automated process performed immediately after the acquisition and evaluation of the quality of information, it is made the most associations possible between objects, their attributes, properties and their rates of pre-determined quality and quality thresholds, considering the existence of two or more data sets available from the same source or from different sources.

The primary fusion meets informational requirements, that is, the criteria of priorities defined the analysis of requirements, such as minimum levels of quality and main properties between information, which is useful to define what should be built first and consequently shown to the human expert. These informational requirements are based on an analysis questionnaire applied to several police officers of different positions, functions, and career time, thus ensuring a heterogeneous view of the subject, and also managing to validate the most important criteria to a given situation.

In the case of the fusion on the demand of the operator, the algorithm is activated once again, but the criteria for integration are entirely selected by the operator via interface, instead of all possible combinations of automatic objects, attributes and properties identified in the acquisition phase.

This association process, now manual, is operated via interface by the expert, and besides being based on objects and attributes, these associations is strongly supported in related quality scores, and suggest hypotheses about information relating to previously identified objects with other objects identified by other sources related to the real environment.

Since this input process is performed by the operator via interface, the criteria for the data fusion process can be chosen and changed by the same operator, who is able to add new features, as well as removing criteria pre-defined by the requirement analysis. This capability demonstrates the flexibility of the framework to receive and process different criteria for a given situation, and allows the agents to better interact with the system based on their experiences and knowledge, thus ensuring a process of construction and development of SAW very close to the real environment.

5 Case Study

The case study was conducted based on procedures and data of the PEMESP, considering emergency calls in São Paulo, Brazil. From the information collected, It was established the informational requirements which define important standards such as the threshold of information quality, the most relevant information to a situation, and the methodology of the decision-making process.

This information enabled the definition of data acquisition methodology, and the main data to be found within a complaint. From the quality threshold and importance of the relationship between information, it was developed the quality assessment techniques. By using all the information collected, the result is the basic ontology of emergency rule in this particular case for theft situations, but ready to be expanded to other crime scenarios.

From the base of ontology defined and the result of requirements analysis, it was developed the fusion engine algorithm, whose input is information with quality scores and semantic meaning, arising from the quality assessment and the ontology. The algorithm presents customizable features and multi-criteria, that is, it accepts several different inputs made by the operator. The algorithm is prepared to deal with the same process in two different modes: automatically, after the acquisition, with the most associations possible between information, respecting the criteria defined in the requirements analysis and in the process on demand, when the operator selects exactly which objects to fuse and what criteria information must be complied during the process and in the outcome. An example of possible association between two objects by synergy attributes is shown in Fig. 7.

Fig. 7.
figure 7

Example association objects

Full size image

As a result of this process, it is obtained the first information with greater consistency, then proving that information is correct on a given situation. This information may contain new information during the process of synergy search and multi-criteria association, which reached levels of quality and property imposed by the process, thus bringing information to the operator.

Before this resulting information is presented to the operator, they once more can be submitted to the fusing process in case they are not within the related criteria or submitted to the quality assessment, which will re-evaluate the quality scores based on new information found. After the quality re-assessment, this information will be set in a new ontology that represents the current knowledge of the situation, and then it will be presented to the operator via interface, such interface is presented by Botega et al. [14].

6 Conclusion

This paper presented a framework for Improving Situation Awareness comprising of a process for multi-criteria fusion of heterogeneous information to mitigate uncertain information about situations and their assets. Such framework may be coupled into situation assessment systems for specialists to reason about information of lower dimensionality and better quality. Hence, this work also presented methods for information fusion, natural language processing, information quality assessment and knowledge representation to be employed in such framework to contribute for SAW. The application of the framework and associated methods generated valid results regarding the obtaining of expected information useful for developing SAW, according to the requirements defined by the domain specialists. Such information was successful incrementally built using syntactical and semantical input. The use of multi-criteria information fusion empowers the assessment of situations by generating several integration possibilities of synergic information for the analysis of a specialist. Also, the specialist has the possibility to define the criteria and the information quality threshold for the parametrisation of the fusion algorithm. As future work, the authors intend to expand and optimize the techniques of acquisition and natural speech processing, as well as methods that make fusion engine, expanding the ability to search and synergistic association data. Also, study and improve the data-mining algorithm from semantic data, as well as the power of its association of multiple criteria, increasing the power to process different inputs, given the multitude of criteria, linguistic or quantitative.