1 Introduction

Endsley (1995) defines Situation Awareness (SA) as “the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future”. The SA model proposed by Endsley is shown in Fig. 1.

The model has three levels (Endsley 2011): (i) perception, which involves the capability to perceive the status, attributes and dynamics of the relevant elements of the environment, (ii) comprehension, which refers to the understanding of what data and cues perceived mean in relation to goals and objectives, and (iii) projection, which relates to the capability of projecting in near future the elements recognized. Endsley’s model is not linear but iterative, with understanding driving the search for new data and new data coming together to feed understanding. Furthermore, it must not be understood as a pure data-driven process since factors such as goals, mental models, attention, working memory, expectations play a pivotal role in SA. In the following we report and summarize some of the key factors of the SA model described in Endsley (2015b) that emphasize other parts of the model:

  • goals and goal-directed processing have a key role in directing attention and interpreting the significance of perceived information; alternating goal-driven and data-driven processing in processing information in the environment is also important;

  • the use of mental models for directing attention to relevant information, providing a means for integrating different bits of information and comprehending its meaning (as relevant to current goals), and allowing people to make useful projections of likely future events and states;

  • pattern matching to schema, prototypical states of the mental model, is such to provide rapid retrieval of comprehension and projection for the recognized situation through critical cues and, in many cases, single-step retrieval of appropriate actions for the situation;

  • a linkage between goals and mental models is such to drive the development or selection of plans and scripts for directing action.

Fig. 1
figure 1

Endsley’s Model (from Endsley 2011)

Full size image

From its conception, several other models of SA have been defined such as Situated SA, Sensemaking, Distributed SA that have been reviewed and analysed by Endsley in comparison to her model in Endsley (2015a, b) concluding that in many cases these models provide explanations that are quite similar. Typically, models of SA foresee the execution of a challenging task, namely Situation Identification or Recognition (Ye et al. 2012) that can be exemplified in a generic workflow composed by three steps: (i) gathering data from a sensor network or other information sources, (ii) deriving more abstract elements from sensor data by including contextual information, and (iii) identifying occurring situations by considering the elements provided by the second step and their relationships. The complexity of these tasks is due to several factors like variety of admissible situations, uncertainty and imprecision of data, dynamic nature of the observed environments and so on. The survey of Ye et al. (2012) reports several techniques for situation identification that are such to deal with the problems above mentioned. Despite that, it is worth mentioning that situation identification is only a phase of SA and processing information according to SA requires a more articulated framework, addressing issues of all the three levels of Endsley model and considering the key factors mentioned in the beginning of the section.

A concrete help comes from the situation theory (Devlin 2006) and by its implementation in formal models such as ontology like in the study by Kokar et al. (2009) or also SAW (Matheus et al. 2003) allowing expressing situations in a common language with a clear semantic and that can be processed by agents. To this purpose, the authors in Jones et al. (2011) have investigated the adoption of agent technology to support traditional human-based SA and proposed a framework with a set of recommendations on how agents can generate, represent and maintain Level 2 and Level 3 SA. On the basis of these recommendations, it is possible to define a hierarchy of agents that, supported by semantic web technologies and ontologies representing contextual information and situations, can cover all the phases of an SA model.

As the reader understands, a computational theory of perception can be very useful for agent-based SA. Zadeh (2001) presents this theory as a way to deal with real-world problems in which decision-relevant information is a mixture of measurements and perceptions. According to Zadeh, perceptions are f-granular, meaning that (1) the boundaries of perceived classes are unsharp and (2) the values of attributes are granulated, with a granule being a clump of values (points, objects) drawn together by indistinguishability, similarity, proximity, and function. Granular computing (GrC) (Pedrycz 2001; Yao and Zhong 2007; Yao 2005; Yao et al. 2013; Salehi et al. 2015) is an information processing paradigm focused on representing and processing basic chunks of information, namely information granules, and finds its origin in the intuition of Zadeh. GrC is today a dynamic research area attracting many practitioners and researchers. Despite that, there are not many applications of GrC to SA and it is quite surprising given the importance that both these research areas give to human-centric information analysis and perception-based reasoning.

In this paper, we aim to pave the way for an integration of these areas via (i) the definition of an high-level view describing how concepts, principles and perspectives of GrC fit into SA model (Sect. 2), (ii) an overview of GrC methods and techniques that can be useful to address some issues of the three levels of the Endsley’s model (Sect. 3), and (iii) a proposal showing the adoption of computational intelligence and Semantic Web techniques to support the development of a multi-agent-based framework for SA aligned with the GrC principles (Sect. 4). The proposal described in Sect. 4 currently covers just methodological and architectural aspects of an SA framework without any claim of completeness. Some relevant aspects, such as evaluation of SA and performance measurements, will be part of future investigations. We conclude the paper with our final remarks and some perspectives on how SA and GRC can be integrated in a systematic way (Sect. 5).

2 Granular computing and situation awareness: the high-level vision

Human understanding and human problem solving involve perception, abstraction, representation and understanding of real-world problems, as well as their solutions, at different levels of granularity. The concept of granularity is pulled by the need for simplification, clarity, low cost, approximation, and tolerance of uncertainty (Yao 2000). In this context, GrC attempts to formally analyse and define methods for granule-oriented problem solving (Yao et al. 2013). Thus, it is easy to understand that GrC is focused on representing and processing basic chunks of information, namely information granules. Information granules are collections of entities that are aggregated together on the basis of their similarity, functional adjacency, indistinguishability or alike. Typically, the process of forming information granules is referred to as information granulation. Granulation of information is a common activity of humans carried out with the intent of better understanding of the problem. Granulation serves as an efficient vehicle to modularize the problem into a series of well-defined subproblems (modules) to reduce an overall computational complexity and computing effort. Moreover, it serves to comprehend the problem and offer a better insight into it rather than get buried in all unnecessary details. In this sense, granulation acts as an abstraction mechanism that reduces an entire conceptual load. By changing the size of the information granules, it is possible to hide or reveal a certain amount of details.

To better understand how granular computation can effectively support SA it is important to describe some basic concepts of granular computing: granules, subgranules, granulation, levels, hierarchies and structures. The descriptions of these elements come from two works (Yao et al. 2013; Yao and Zhong 2007), which are well known by GrC researchers.

As already affirmed, the basic elements of granular computing are granules. A granule may be considered one of the small particles forming a larger unit. For instance, a granule can be a subset of a set, a class of objects, a section of an article, and a module, a component or a service of a system. Granules can be decomposed into smaller or finer granules called subgranules. To construct or decompose granules we need to employ a specific two-way operation called granulation. Granulation is recognizable also in problem solving processes when humans decompose a complex problem into less complex subproblems or compose the solutions of subproblems into the solution of the whole problem. On one hand, construction is related to the process of forming a larger and higher level granule from smaller and lower level subgranules. On the other hand, decomposition represents the basis of the process of dividing a larger granule into smaller and lower level granules. The former is a bottom-up process, and the latter is a top-down process.

Granules and subgranules can be organized by means of levels, hierarchies and granular structures. Levels consist of one or more granules that are formed with respect to a particular degree of granularity. Granules, within a level, are defined and formed within a particular context and are related to granules in other levels. A granule captures a particular aspect, and collectively, all granules in the level provide a granulated view. Granules in different levels can be linked by different types of relationships (e.g. ISA, partial ordering, refinement/coarsening) and operations (e.g. granulation) on them. The ordering of levels can be described by the notion of hierarchies. Furthermore, it is interesting to analyse the internal structure of a granule. Such structure provides a proper description, interpretation, and characterization of the granule and can be represented as a hierarchy consisting of many levels. Granules have at least three main properties: internal properties, external properties, and contextual properties. A granule can be observed as both a collection of individual elements (internal properties) and an inseparable whole (external properties). Moreover, the existence of a granule is meaningful only in a given context (Yao 2006). In what follows, we describe three different perspectives related to the application of GrC to SA systems (Sect. 2.1).

2.1 Three perspectives of GrC for situation awareness

Granular computing can be analysed by considering three different perspectives (Yao 2005). It can be viewed as: (i) a general method of structured problem solving, (ii) a paradigm of information processing, and (iii) a way of structured thinking. From the point of view of this paper, it is interesting to note that the three perspectives can be considered to define solutions for different aspects related to the design of systems for SA.

In fact, according to the study by Jones et al. (2011) which introduces the concept of agent-based systems supporting decision making processes and human situation awareness and Endsley (2011) which proposes a methodology to design systems for situation awareness, the first perspective is useful when we consider the experience of a human operator interacting with a system for situation awareness; the second perspective could be considered during the design of such systems, and lastly, the third perspective is particularly useful when we are defining situation identification tasks (performed by the aforementioned agents) and we need several algorithms to granularize information.

Fig. 2
figure 2

Three perspectives of GrC and situation awareness

Full size image

Figure 2 shows how the three perspectives of GrC can be mapped into the different phases of the life cycle of a system for situation awareness. In particular, the first perspective is used when designers and domain experts execute the design tasks that provide a first granularization of the domain according to the goals of the systems. The results of this phase are exploited by the second phase that considers the second perspective of GrC. In this phase-specific techniques of Computational Intelligence are employed to concretize the information pyramid and realize the granulation of information. The third phase considers the deployment of the constructed hierarchy of granules to support human situation awareness. As shown in Fig. 2 these phases can be associated with the phases of a common process for information systems engineering, called Systems Development Life Cycle (SDLC) (Alexander and Maiden 2005).

Fig. 3
figure 3

Information pyramid

Full size image

2.1.1 A paradigm for information processing

According to its computational perspective, GrC focuses on a paradigm for representing and processing information in a multi-level architecture. Information granules exhibit different levels of granularity. Depending upon the problem at hand, it is possible to group granules of similar size (that is granularity) together in a single layer. If needed, smaller granules are analysed and arranged in a higher level. Thus, such granules are arranged in another layer. In total, the arrangement of this nature gives rise to the information pyramid. Information granularity implies the usage of various techniques that are relevant for a specific level of granularity (Pedrycz 2001) as reported in Fig. 3. The bottom level (that is the level where we can find the highest level of granulation in terms of number of granules) is typically associated with numeric processing. The intermediate level provides larger information granules. Lastly, the top level (usually associated with the lowest level of granulation) is usually devoted to symbol-based processing. The selection of the right techniques at each level is driven by the specific domain, available resources, requirements, and so on. Among these techniques, the most relevant ones associated to GrC will be reviewed and classified in Sect. 3.

The information pyramid is what we find when dealing with situation identification task (Ye et al. 2012). Several works in this field evocate the concept of information pyramid. Two of these are reported below.

The authors of the study by Mittal et al. (2012) provide an approach to infer knowledge on situations in a physical environment, equipped with sensors, by exploiting three levels of computation. Sensors (light sensors, microphones, GPS, biosensors, body temperature, etc.) and logical sensors (time of the day, schedule of the day, universal or known facts, etc.) observe features of the world and provide data positioned at the lower level of the pyramid. Sensor data become inputs for the intermediate level where context information are deduced and positioned. Here, context is considered as any information about user, his/her environment or activities. For instance data coming from accelerometers (at the lower level) provide motion context as whether a user is walking, sitting or running. Context information is derived using methods like Fuzzy Logic, Probabilistic Logic, Bayesian networks, Hidden Markov models, Dempster–Schafer theory of evidence, Rule-Based Reasoning and Ontological Reasoning (Ye et al. 2012). At the top level, a number of context information, occurring within same time frame, are transformed in abstract actionable objects, namely situations. This hierarchy of data organization increases the usefulness of data and decreases its size. The approach is sketched in Fig. 4a.

Fig. 4
figure 4

Information pyramid in situation awareness

Full size image

The concept of information pyramid is also considered in Kokar et al. (2009) where the authors introduce four planes of abstraction by considered both the Endsley’s Model (Endsley 2000) and the JDL Data Fusion Model (Hall and Llinas 1997). The bottom plane is the physical world whose aspects can be monitored. The next plane is that of perception where we can find the representation of objects of the physical world that are observed through sensors. The next plane represents situations, i.e. the knowledge on all objects in a specific area. At the top plane, we can find projections, which are symbolic information to anticipate future events and their implications. The multi-plans framework is depicted in Fig. 4b.

2.1.2 A method of structured problem solving

GrC promotes a general approach for describing, understanding, analysing, examining, and solving real-world problems based on a multi-level structure through abstraction, control, complexity, detail, resolution, and so on. The generation and utilization of hierarchical organizations and structures, at a more practical level, rely on a strategy known as “divide and conquer”. With this strategy, a problem described with larger granules is decomposed into a family of subproblems (top-down) described with smaller granules, and the solution of the problem is obtained by combining the solutions of subproblems (bottom-up).

According to the need to support the situation identification task by adopting an information pyramid, it is clear that both top-down and bottom-up directions of granulation are relevant. In fact, if the bottom-up granulation is needed to transform sensor data (low-level granules) in coarser units (middle-level granules) and these units in more abstract elements (high-level granules), until reaching high-level granules (situations or projections). On the other hand, the top-down direction of granulation can be useful to support the design phase of a system for SA according also to the methods of structured problem solving. A concrete idea of such a process is identifiable in the Goal-Directed Task Analysis (GDTA) proposed by Endsley (2011). GDTA is a form of cognitive task analysis and focuses on the goals the human operator must achieve and the information requirements that are needed to make appropriate decisions. Information is, step-by-step, decomposed until reaching finer elements that cannot be further decomposed. It is important to underline that GDTA focuses on dynamic information requirements rather than static system knowledge, i.e. the approach considers the information, needed to perform well a specific task, that has to be acquired and analysed by the operator in a certain domain during the execution of such task. The needs for this information are called SA requirements.

The result of GDTA is an abstract structure (see Fig. 5) establishing what are the low-level data to be elaborated to obtain higher level information. The computational techniques allowing this elaboration at different abstraction levels are matter of the previously analysed perspective.

Fig. 5
figure 5

Goal, decisions and SA requirements resulting from GDTA

Full size image

2.1.3 A way of structured thinking

Hierarchical organizations and structures exist largely in the real world. It is possible to find them in many natural, artificial and man-made systems. Human perception and understanding of the real world mostly depends, to a large extent, on such nested and hierarchical structures. GrC, as structured thinking, explores multi-level granularity that exists in the physical world. This philosophy is consistent with, and nicely reflects, reality. GrC helps us to arrive at accurate and natural description, as well as in-depth understanding, of the inherent structures and complexity of the real world.

This perspective of granular computing helps to design systems for SA, and in particular their user interfaces, that support well the human operators in interacting with the information (more or less abstract) related to the environment and relevant for the tasks they are involved in. The availability of hierarchies, levels and granule structures supports the definition of user interfaces sustaining human understanding of situations. Human operators could start from the higher level of granules and go down until a clear comprehension of the situation is achievable, by exploiting operations like, for instance, zooming in and zooming out (specialization/generalization, more details/less details and so on).

3 Granular computing and situation awareness: the low-level vision

This section presents an overview of GrC to support different aspects and phases of SA. With regards to GrC we refer to the categorization of research focus areas presented by Salehi et al. (2015) that is based on a systemic mapping of the most recent contributions in GrC (January 2012–August 2014). The categories are shown in the following figure from Salehi et al. (2015) (Fi g. 6).

Fig. 6
figure 6

Research focus area categories in GrC from Salehi et al. (2015)

Full size image

Data analysis techniques are devoted to represent information granules for spatiotemporal and heterogeneous data. Two mainstreams of these techniques have been categorized by Salehi et al. (2015): Human-Centric Way (HCW) dealing with data representation interpretable by humans and Interval-Based Evolving Methods (IBEM) dealing with heterogeneous stream of data in time-varying systems. Concept formation and learning involve the adoption of learning strategies to draw correspondences between granules (and their relationships) and concepts (and their relationships). Interaction deals with discovering and modelling interactions of objects in interactive granular systems. Segmentation, lastly, deals with a set of approaches suitable for partitioning data (video, images, signals, text) and in some cases classifying the segments. Additional information of these categories can be found in the study by Salehi et al. (2015).

Figure 7 proposes a tentative mapping of the above categories on the Endsley’s model of SA.

Fig. 7
figure 7

GrC techniques and SA

Full size image

Before presenting an overview of GrC techniques suitable to address main issues of the three levels, we emphasize some key aspects of the Endsley’s model that motivate our mapping:

  • the SA requirements are different for different domains and human roles (a pilot has to perceive elements that may be different from the ones of an air controller) and demand for an alternation of data-driven and goal-driven processes in processing information of the environment (Endsley 2015b). We argue a granular approach to data analysis is such to satisfy these requirements by including in fusion processes' heterogeneous and spatiotemporal streams of data as well as human-oriented information such as for instance judgments (Kaburlasos and Pachidis 2014).

  • it is recognized that most of the problems with SA occurs at the level 1 because of missing information or information perceived in a wrong way (Endsley 2011) or also information not pertinent with respect to the specific goal. In the Perception level, key tasks relate to object classification and state estimation. We argue a granular approach can solve some of the issues traditionally associated with this level:

    • first, the principles of SA-oriented design (Endsley 2011) demands to organize information around goals and provide a proper level of abstraction of information (true also for level 2). Grouping information depends thus on the goal, domain requirements and knowledge, mental model. The concept of information granules suits well with these requirements and the issue here is to design information granules that are representative of an objective and a domain, and specific enough to support easy interpretation at the appropriate level. Pedrycz and Homenda (2013) proposed the principle of “justifiable” granularity as a way to design information granules on the basis of experiential evidence so that the granules are justified with respect to experiential data and specific enough to be meaningful. In Pedrycz et al. (2015) a parametric version of this principle is defined that includes an additional interesting aspect for SA, i.e. “involvement of inhibitory experimental evidence (viz. data that have to be excluded for the constructed information granules)” and is adopted to define a collection of meaningful and interpretable descriptors that, as the authors argue, are a first step towards classification and prediction;

    • filtering of extraneous information (not relevant to SA) and reduction of data is beneficial to SA to provide a correct classification and understanding of the relevant elements of the environment. This is another principle of SA-oriented design (Endsley 2011) but its application is challenging due to the characteristics of real-world data (imprecise, vague, uncertain) and the wide number of different data sources (e.g. sensors). GrC can deal with these issues and segmentation techniques provide good support for this aspect. An issue is to filter proper information and sources on the basis of objectives and goals. For example, Peters et al. (2001) propose the adoption of Rough Set Theory and rough integrals to select the most informative sensors for the specific of the fusion system, such as object classification.

  • Obtaining comprehension of the current situation is the key issue of level 2. Endsley (2015b) emphasizes the importance of pattern matching to schemata, i.e. prototypical states of the mental model, to provide rapid retrieval of comprehension and projection for the recognized situation. Also scripts, sequences of key actions associated to each schema, are useful for this level. To achieve this, the main task is the recognition of relevant elements and their relationships. The GrC techniques categorized under concepts formation and learning and Interactions support this level. In the work by Skowron et al. (2012) authors propose an approach “aimed at inducing compound granules relevant for solving problems such as approximation of complex concepts or selecting relevant actions (plans) for reaching a target goal”. Based on the rough set theory, authors use the concept of approximation spaces that are fundamental granules used in searching for relevant complex granules called as data models, that approximate more complex concepts and their relations. Authors present also interesting examples of strategies for the extension of approximation spaces from samples of objects (that can be relevant elements of level 2) onto a whole universe of objects (that can represent a recognized situation of a schemata).

In the following subsections we present an overview, with no claim of completeness, of methods and techniques that can support the three phases of SA (Perception, Comprehension and Projection) considering, in many cases, methods and techniques not surveyed by Salehi et al. (2015). The criteria for selecting the methods and techniques are based on analysis of SA requirements that arise from the principles of SA-based design proposed by Endsley (2011).

As a summary, we anticipate that most of the methods and techniques analysed in our overview can support the first level in addressing issues related to a proper organization of the information around goals and objectives, i.e. building information granules, and to object assessment and state estimation via techniques of outlier detection, attribute reduction and data conflict resolution. For the second level, we focused our attention on techniques that can support in understanding the right meaning of the elements perceived and, thus, mainly techniques of concept formation. In the third level, we limited to an analysis of techniques supporting prediction in a future state of some elements of a recognized situation.

3.1 Perception

The Perception level of the SA model is devoted to perceive and recognize elements of the environment by combining observations and measurements from different sources. Despite the fact that GrC enables a human-centric way of information processing that is a key aspect of the SA, there are not so much applications of GrC to situation observability. We argue two main issues related to the Perception level can be solved with GrC: designing information granules around goals and support information filtering via analysis of outlier, conflicting and spurious data and attribute reduction.

The principle of justifiable granularity (Pedrycz et al. 2015) previously mentioned allows construction of granular descriptors according to linguistic characterization and two measures, coverages and specificity, with the first concerned with the ability to represent (cover) an experimental data set and the second related to a level of abstraction conveyed by the granule (Pedrycz 2015). Fuzzy clustering, in particular Fuzzy C-Means, is employed as a vehicle to build information granules. The problem of successive refinement and generalization of prototype information granules is discussed in Balamash et al. (2015).

Construction of information granules according to the principle of justifiable granulation fits well with the SA requirements of selecting the proper information and at the right level of abstraction for the specific goal and objective. These approaches work well when there is the availability of experimental evidence resulting from previous and similar situations that can be stored in schemata and scripts. Moreover, the principle of justifiable granularity is applicable also to time series (Pedrycz et al. 2014) and, in a former proposal, applied to signal analysis (Pedrycz and Gacek 2002). With regard to signal processing, another interesting perspective (always based on the justified granularity) is the hybrid method based on neural networks, GrC and evolutionary computing proposed in Gacek (2015). In both the cases, the works refer to medical domain and we did not find an interesting application of this principle to traditional domains of SA.

Recently, Sanchez et al. (2015) have proposed an approach for construction of information granules based on the theory of the uncertain that can be useful in environments characterized by high level of uncertain and noise, such as sensor networks. The basic idea employed in the work is that a reduction of uncertainty can be obtain by the difference of two uncertain models of the same information, e.g. a priori and a posteriori models. In building information granules, uncertain is evaluated for a first sample of information in the form of a type 1 Fuzzy Gaussian membership function, and a similar evaluation of uncertain is done for a second sample. The difference between the two membership functions create the Fingerprint Of Uncertain (FOU) and a type 2 fuzzy set is used to form the granule; in case of no difference type 1 is adopted.

An approach to support SA in life support systems has been proposed by Drayer and Howard (2012a, b) and is based on the creation of sensing spaces that are subsequently decomposed into perceptual elements or granules via the adoption of a perception function that is a Fuzzy Associative Memory. Also in this case, the criteria for construction of granules is based on human assessment and these human-input datasets are transformed in granular structures using particle swarm optimization, and then adjusted to satisfy a coherence measure based on the Ruspini’s condition (Ruspini 1969).

A number of GrC techniques can be employed to support recognition of elements, information filtering and attribute reduction requirements of SA. Some recent methods that can support recognition of the elements of an environment are employed with Granular Neural Network. It is the case for instance of the method proposed by Sánchez et al. (2015) that has been tested with human recognition based on the face biometric measure. The method is based on the adoption of a modular neural networks optimized with a hierarchical genetic algorithm, and GrC is used to split the whole database into sub modules. Granular neural networks have been employed also for classification of land use/cove images (Meher and Kumar 2015), and for fusions of numeric and linguistic data (Zhang et al. 2000), and this last case appears of interest in several SA scenarios where important sources of information can be textual (e.g. social media).

With regard to filtering the most relevant data for fusion and classification objectives, we already mentioned the work of Peters et al. (2001) on the adoption of rough integrals to select the most informative sensors for a specific objective. A similar approach is employed in Haijun and Yimin (2006), and an hybrid approach combining fuzzy and rough set for classification under uncertainty is presented in Guan and Feng (2004).

Analysis and detection of outlier and spurious data are investigated in several works by Jiang et al. (2005), Nyuyen (2008), Shaari et al. (2009), Chen et al. (2010) and Jiang and Chen (2015). A commonality among these works on outlier detection is the adoption of Pawlak theory of rough sets and its capability of approximating sets, via lower and upper approximation functions, to detect outlier objects having abnormal attributes and properties (generally in boundary regions). In the work by Shaari et al. (2009) is used the concept of Non-Reduct to discover a set of attributes that may contain outliers, Chen et al. (2010) proposes the adoption of outlier detection algorithm based on the neighbourhood rough set model, Jiang and Chen (2015) introduce the concept of GR-based outliers and proposes a detection algorithm working on this concept. With regards to spatiotemporal requirements, a specific application for detecting spatial and temporal outliers is proposed in Albanese et al. (2014).

Attribute reduction plays a key role in applications requiring SA since high-dimension data are common and this requires computation time and space. Several works propose rough sets and GrC methods to solve the attribute reduction problem but few of them indicate or present applications to specific problems of SA. An application of rough set theory for attribute reduction to support situation recognition via classification of precursory information in reference to earthquake rupture analysis is proposed in Dutta et al. (2013). Starting from the consideration that most of these methods concentrate on data only, posing a difficulty in choosing appropriate attribute reducts for specific applications, recently Jia et al. (2015) propose a generalized framework allowing human expert to specify conditions in terms of group of measures and thresholds which are relevant to user requirements or real applications. The proposed framework gives the possibility of choosing the appropriate reducts on the basis of users and application requirements, and this can support the goal-oriented information processing principles of SA-based design.

An issue that arises in concrete applications demanding SA is resolving data conflict, e.g. when we have multiple values of an observation or a variable that are not compatible. Resolving the data conflict issue is necessary to achieve a proper perception of elements. In Yager (2004) it is presented a multi-sensor data fusion framework including voting-like process to resolve conflict among data using a measure of compatibility. The framework allows characterization of user requirements and needs such as level of abstraction and results in granular and multi-granular objects as fused values. An alternative approach to the voting process can be the adoption of soft-consensus model supporting human-like perception processes (Herrera-Viedma et al. 2014).

3.2 Comprehension

The second level of the Endsley’s model is devoted at understanding what data and cues perceived in the first level mean with respect to goals and objectives. Comprehension is achieved via a meaningful integration and prioritization of the elements of the environment perceived in the Perception level. Endsley (2011) evidences that errors in this level typically happen because human operators are such to see and hear the data and elements of previous level but are unable to correctly understand the meaning of this information. GrC for concept formation supports comprehension presenting information required to this level. Based on the triarchic theory of GrC, the approach described in Yao et al. (2013) proposes two strategies for concept learning, namely, an attribute-oriented strategy for searching a space of partitions and an attribute value-oriented strategy for search space of coverings. A perspective focused on cognition of concept learning via GrC is analysed in Yao (2009) and Li et al. (2015), and proposals for building tools for automatic understanding of data via granular cognitive maps are presented in Homenda and Pedrycz (2015).

Other approaches leverage of Formal Concept Analysis (FCA) and concept lattice. It is the case of Singh et al. (2015), where the authors propose an algorithm for generating interval-valued fuzzy formal concepts using the properties of interval-valued fuzzy graph and Galois connection and incorporation of interval-valued fuzzy graph to the concept lattice. The authors in Wu et al. (2009) address the issue of knowledge reduction in formal concept analysis via the adoption of GrC and, specifically, via the concept of granular reduct of a formal context (i.e. a minimal attribute set preserving the object granules of a concept lattice obtained from a full attribute set). These approaches can be useful if formal contexts of the situation to recognize are available in schemata or scripts.

We already discussed the adoption of Granular Neural Network for object classification and recognition in the previous subsection. In literature, we have found also an application of similar methods for situation recognition in a multi-agent framework devoted at enabling SA. It is the case of Castellano et al. (2014) that propose the adoption of fuzzy rules used by agents for situation recognition, and the adoption of neuro-fuzzy learning to adjust the parameters of the rules.

In the study by Jankowski et al. (2013) and Skowron and Jankowski (2015) is defined the approach of Interactive Rough Granule Computation as a way for modelling interactive computation with rough set and other soft computing approaches. The concept of Complex Granule is defined that allows to link traditional information granules to physical objects, and can be used by agents to make decision via adaptive (intuitive or rational) judgment. Via interactive hierarchies of complex granules, authors evidence how it is possible to approximate vague and complex concepts that can be relevant in processes of situation recognition and decision making, such as safe driving in traffic control applications.

3.3 Projection

This level is devoted to project in the near-future elements of the situation recognized in the previous level. Some models useful for the projection step of SA include time series and regression based (Fricker 2013). Description and prediction of time series has been deeply investigated in GrC, and we mention just few recent works. The authors in Al-Hmouz et al. (2015) propose a framework in which information granules are based on time windows, amplitude and change of amplitude, and employ fuzzy relations to predict amplitude and its change. In Wang et al. (2015) the issue of long-term prediction is addressed via the development of a forecasting model combining a modified fuzzy c-means and information granulation. Authors present also application of their model for forecast of power demand and daily temperature. In Lu et al. (2014) authors employ fuzzy cognitive maps to describe granular time series (built with fuzzy c-means clustering algorithm) and perform predictions.

4 Proposed approach

In this section, we describe our proposal for the adoption, from a methodological and architectural perspective, of computational intelligence techniques to support some SA aspects. The whole framework for SA (originally introduced in D’Aniello et al. 2015c; Benincasa et al. 2015) is sketched in Fig. 8. It consists of four layers which de facto realize the information processing pyramid introduced in Sect. 2 (aiming at transforming raw sensor data perceived from the environment in situations which support decisions and projections in near future) and by a traversal cognition layer that supports the information processing flow. In the following subsections, we analyse the framework from a methodological and architectural perspective. Moreover, we propose two examples based on our previous research findings in Sect. 4.2. A complete case of study involving a whole SA process, including measurements and performance assessments, is part of our future research activities.

4.1 Improving SA with multiviews and multilevels

From a granular computing point of view, the framework can be considered as an information processing pyramid consisting of multiple levels of granulation. This can help in processing information of real-world complex problems for which is crucial to choose suitable and accurate representations of the problem. Along with this strategy, a problem (such as making human operators aware of the current situation) which is described with coarse granularity can be decomposed into a family of subgranules described with smaller granules. The solution of the problem is obtained by combining the solutions of the subproblems. Moreover, for the same problem, it could be possible to view it from many different angles, and associate a representation with a particular view. A representation can make certain features explicit and hide the others. Multiple views for the same problem could be exploited to avoid the limitations of a single view representation. Moreover, for each view it is possible to consider multiple levels of abstractions, which represent the problem at a particular level of details. Thus, a crucial task is to define criteria for evaluating the selection of levels of abstraction and views at which searching the most suitable solution. This is especially needed, for instance, when more resources are available, or when new requirements are given, it is possible to search solutions at further levels of accuracy or even using a different view (Yao 2010). Handling representations with multiple views and multiple levels allows the definition of adaptive systems, which can autonomously switch among views and, in the same view, among levels. This switch can be implemented by pre-planning all the plausible representations and defining the rules for changing levels or views. An alternative switch approach is that the system learns in specific contexts what are the most suitable couples constituted by views and levels. This can be accomplished by employing a feedback-based learning algorithm.

The proposed approach, depicted in Fig. 8, targets this multiviews and multilevels approach to Situation Awareness. In particular, the highest level of granulation (mainly associated with numerical data) is the Sensor Data Pre-processing layer. In this layer, raw data gathered by sensing devices deployed in the environment are pre-processed by means of data-cleaning techniques, outlier detection, filtering, and so on. In the next level, namely Generating Observations layer, these data are processed and aggregated to produce observations: an observation represents the current value of an environmental phenomenon which is of interest for the current operators’ goals and it is represented in a human-understandable format. For instance, it can be represented by a fuzzy linguistic term set or an ontological concept. Different expert-based techniques can be adopted to produce observations. The most common ones are Fuzzy Cognitive Maps (see Sect. 4.2.1 for an example of this technique) and Dempster–Shafer Theory of evidence. Besides the data received by previous layer, these techniques can also exploit contextual information and background knowledge (contained in the Cognition layer) to produce more meaningful pieces of information. Such observations are further processed and aggregated to identify situations, which can be seen as the information granules of the next layer (i.e. Situation Identification layer) with a lower information granularity. Situations represent the state of the environment from the point of view of the operator and are thus related to a specific goal and to a specific view. A situation is identified by considering the current observations and by identifying the relations among them and with the objects (both physical and abstract) of the environment. The identification and exploitation of these relations allow for identifying the situation, which can be represented, for instance, by an ontological concept. As for the previous layer, several techniques can be used for situation identification (Ye et al. 2012). A plausible approach is to use rule-based techniques in which the relations on the observations represent the antecedent of the rule, while the situation represents the consequent of the rule. Such rules can be both defined by a domain expert or learned by means of machine learning techniques applied on training sets. Lastly, the situations are processed (and, in some cases, further aggregated) to support decision making. This happens in the last layer, the Situation Projection layer, which is characterized by the lowest level of granularity. Here the operators try to predict the evolution of the environment by projecting the situations in the near future, to anticipate such evolution and make decision according to the effects of the identified situations.

Fig. 8
figure 8

Proposed approach

Full size image

The proposed approach can be implemented by means of multi-agents technologies and Semantic Technologies. Indeed, the agent-oriented paradigm, together with the Semantic Technologies of the Cognition layer, endows a flexible behaviour to the whole framework, fostering its extensibility and instantiation to different application domains, allowing for the adoption of different computational techniques for processing information and supporting decision making (Castellano et al. 2014). In particular, Semantic Technologies and, in particular, ontologies, can be exploited for sustaining the information processing in each of the four layers of the framework. More in details, the Semantic Sensor Network Ontology (SSNO) can be employed for describing information on the sensor network. Each sensor can be described by means of a set of metadata (including sensor type, measurement types, quality of service, etc.) which can be exploited (in the first layer of the proposed approach) to perform data processing. In the Generating Observation layer, ontologies can be used to assign meanings (by connecting them to events and, above all, situations) to the generated observations. In the Situation Identification layer, ontologies can be used for describing contextual information that can be helpful for deriving additional attributes, mostly related to common knowledge, which can be considered for identifying a situation.

Lastly, the choice to employ a multi-agent paradigm allows implementing the different level of granulation that are required for each specific problem, by selecting the most suitable computational intelligence techniques depending on the specific domain of application. In this scenario, the Semantic Web technologies (e.g. ontologies) provide the framework to make the heterogeneous agents operating and cooperating, while the multi-agent paradigm provides a dynamic elastic layer that works as a glue among sensors (D’Aniello et al. 2015c).

4.2 Concrete examples

In this section, we describe two examples related to the application of Computational Intelligence techniques to address two relevant issues of Situation Awareness: generating observations and situation identification. The examples reported relate to the above-mentioned methodological and architectural perspective.

4.2.1 Generating observations via fuzzy cognitive maps

Fuzzy Cognitive Maps (FCM) provide a structure allowing qualitative reasoning on the state of complex systems (D’Aniello et al. 2015c). Briefly, an FCM is a formal graph model of a system described in terms of concepts (the nodes of the graph) and connections (the directed edges of the graph) (Pedrycz and Homenda 2012). Such connections represent cause/effect relationships among concepts, and the strength of the connections is quantified by numeric values in the [−1, 1] range. Positive values describe a positive causal interaction between the two nodes, whilst negative values represent inhibitory character of the connection. The dynamics of the map (i.e. how the values of the concepts are computed) is described by the following equation:

$$\begin{aligned} A^t_i = f\left(\sum _{j=1, j\ne i}^{n}A^{t-1}_{j}W_{ji}+A^{t-1}_{j}\right). \end{aligned}$$
(1)

where \(A^t_i\) is the value for the ith concept at time t, \(W_{ji}\) is the weight of the edge between the ith concept and the jth concept and f is a nonlinear continuous nondecreasing function assuming values in the unit interval. The function f produces a level of activation of the ith concept normalizing it to the [0,1] interval (Pedrycz and Homenda 2012).

In the proposed framework, we use an FCM for generating the value of an observation. In particular, let us consider that an observation agent obs is committed to monitor the current value of an environmental phenomenon \(\varphi \) by generating the value for the observation \(O(\varphi ).\) Thus, the role of the agent is to express its opinion on this phenomenon by selecting one of the possible values from the set of admissible opinions \(X={x_1, \ldots x_n},\) \(n\ge 2\) for this phenomenon. At each instant t, the opinion selected by the agent obs will represent the value for the observation \(O(\varphi ).\) More formally, let \(S={s_1, s_2, \ldots, s_h}\) be the set of sensing agents that gather data about the phenomenon \(\varphi. \) The behaviour of the observing agent \({\rm obs}_1\) can be modelled as the function \(\sigma ({\rm sd}^{T}_{s_1}, \ldots, {\rm sd}^{T}_{s_h}, C^{ T}) \rightarrow (x_1, \ldots, x_n)\), where \({\rm sd}^{T}_{s_k}\) \((1 \le k \le h)\) are the data (list of records) provided by the sensing agent \(s_k \in S\) in the time slice T. In addition, C is a set of context information valid in T and \(x_j\) \((1 \le j \le n)\) is a value in [0, 1] and it indicates the preference degree of the agent obs for the opinion \(x_j\) \((1 \le j \le n)\) included in the set X for the phenomenon \(\varphi _1.\)

Fig. 9
figure 9

Behaviour of an observing agents which uses FCMs for generating its opinions on the observed phenomenon (from D’Aniello et al. 2015c)

Full size image

The behaviour of each observing agent consists of an FCM for each alternative in X. In this map, the value of each concept in the last layer (output layer) of the map, represents the preference degree for the corresponding opinion, allowing us to determine the value for the phenomenon (the opinion with the highest preference degree will be selected as the value for the observation).

Let us consider the following example (taken from Perera et al. 2012) and presented also in D’Aniello et al. (2015c) to explain the behaviour of an observing agent. The observing agent obs has to monitor the phenomenon \(\varphi =\)“health of crops” and assume that the set of possible alternatives is \(X=(x_1=\)“infected by Phytophtora disease”, \(x_2=\)“not infected by Phytophtora disease”). Moreover, a set of sensors in the environment monitor air humidity, air temperature and leaf witness. Such data are pre-processed by three different sensing agents. The output of each sensing agent is represented by the value for the linguistic variable {LOW, HIGH} related, respectively, to the the three fuzzy sets AirTemperature, AirHumidity and LeafWitness. Figure 9 shows how raw data are analysed by sensing agents and then the calculated fuzzy values are used by the observing agent as input for its two FCM to generate its opinions on \(\varphi. \)

4.2.2 Identifying situations via neuro-fuzzy network

The task of identifying the current situations (carried on by Situation Identification agents) starting from the generated observations, can be accomplished by means of different techniques. Roughly speaking, when the observations are formally represented (e.g. by means of ontological concepts or fuzzy linguistic term sets), it is possible to use both learning-based approaches, which try to associate the set of occurring observations to one or more situations via machine learning techniques, or specification-based approaches in which an expert is committed to define the rules for inferring a situation from the generated observations (e.g. using ontology-based inference). An example of a machine learning technique for identifying a situation can be found in Benincasa et al. (2015), in which an approach based on Fuzzy Formal Context Analysis has been exploited to identify the relations among observations (described as fuzzy sets) and situations. In D’Aniello et al. (2014, 2015b), instead, approaches based on ontological reasoning have been proposed.

In this paper, we propose an approach based on Neuro-Fuzzy Network (an example of such a technique applied in a blended commerce scenario can be found in D’Aniello et al. 2015). A neuro-fuzzy network (or neuro-fuzzy system) is a “system similar to a fuzzy controller where the fuzzy sets and rules are adjusted via neural networks tuning techniques in an iterative way” (Vieira et al. 2004) using datasets containing input and output system data. In the learning phase, a neuro-fuzzy system acts de facto as a neural network that learns its internal parameters whilst, at run time, it behaves as a fuzzy logic system. The combination among fuzzy logic and neural network provide us with the advantages of both techniques, thus to limit their weaknesses. Specifically, the obtained system benefits from the capability to learn, which is typical of the neural networks, and from the easiness in interpreting the results of a fuzzy systems and from its robustness in relation to the possible disturbances in the system.

In the proposed approach, with respect to the classification proposed by Vieira et al. (2004) and to the definition provided by Nauck et al. (1997), we adopt a Hybrid Neuro-Fuzzy System, which is a fuzzy system that uses an heuristic learning strategy based on neural networks to determine its parameters (i.e. fuzzy rules and fuzzy sets). Furthermore, we exploit the a priori knowledge about the specific situation identification problem to define the set of fuzzy rules, and then a neural network learning technique is applied to learn only the parameters of such rules (i.e. the learning algorithm is not applied to learn the rules but only the parameters of such rules).

Fig. 10
figure 10

Neuro-fuzzy network

Full size image

Figure 10 shows an example of the proposed neuro-fuzzy system. It follows the Adaptive Network-based Fuzzy Inference System (ANFIS) architecture (Jang 1993), which is characterized by five layers and it implements a Takagi Sugeno fuzzy inference system. The first layer represents the input of the system. In our approach, each node of this layer represents the value of an observation \(\rho _i.\) The second (hidden) layer represents the fuzzy sets defined for the input variables, which also represent the antecedents of the fuzzy rules. Each node in the third (hidden) layer represents a rule: it is connected with the nodes of layer two, representing the antecedent of the rule, and with a node of layer fourth, which represents the consequent of the rule. Each node of this layer normalizes the strength of the correspondent rule (by a factor that is learned via the neural network). In particular, the Fig. 10 implements the rules reported in Table 1.

Table 1 Rules for identifying the situation implemented by the neuro-fuzzy system of Fig. 10
Full size table

Such rules, in the context of a blended commerce scenario, can be interpreted as the rules for identifying the so-called situation of interest, which represents the situation in which a number of customers in a shopping mall are interested in a specific product. It is identified by considering two observations: (i) \(\rho _1,\) the attention time (i.e. the time customers spent in front of a product) and (ii) \(\rho _2,\) the aggregation (i.e. the number of people in front of a product or a showcase). Further details on such scenario can be found in D’Aniello et al. (2015).

Let us consider again the neuro-fuzzy system of Fig. 10. In this figure, each node of the fourth (hidden) layer represents a consequent of each rule. Lastly, the fifth layer calculates the output of the system as the sum of the values of all the rules and it performs the defuzzification process. Each node of the output layer represents a situation.

The advantage of this approach relies in the capability to adapt the parameters of the second and fourth layers via supervised learning methods (Jang 1993). This allows to adapt membership functions to different contexts in which the same rules can be applied but, due to substantial differences in contextual and environmental characteristics, different memberships of the variables need to be considered. Let us consider again the blended commerce scenario as an example. In this case, it is useful to consider different membership functions for the aggregation parameter in the above neuro-fuzzy system, to take into account that the number of people which crowds the shopping mall is different on a working day or on a Saturday afternoon.

5 Conclusions and future works

GrC and SA are two dynamic research areas attracting many practitioners and researchers. GrC is at the basis of human perception and of a computational theory of the perception, and it is easy to recognize its importance in SA models. Furthermore, as we show along this paper, GrC and SA share some important concepts and principles and GrC techniques can be adopted to solve open issues in the three levels of the Endsley’s model. Despite that, the two research areas are still today considered as silos. What is missing is a systematic approach to study interactions and correlations between GrC and SA. We recognized this gap and our paper is a first result in this direction. We evidence two correlated perspectives for this systematic study.

Regarding the first perspective, we paved a way in this paper. It relates to a comprehensive adoption of the principles, concepts, methods and techniques of GrC to enforce SA and in particular agent-based SA. We started with an high-level view of GrC and SA, moved along an overview of the techniques that can be adopted to enforce SA and concluded with some methodological and architectural examples of Semantic Web and Computational Intelligence techniques supporting our view. What we are working on now is the definition of an overall framework to enforce SA with GrC. In this framework a key role can be played also by the Computing with Words paradigm of Zadeh that can support computation and reasoning on natural language statements representing humans or agents perceptions of the reality of an environment. A second perspective relates to the other direction, that is leveraging on SA as an information feedback for GrC and granulation process. As an example, it could be interesting to understand how data, information, knowledge, acquired on the basis of SA, can support GrC and, in particular, its principles of justifiable granulation, thus to support in choosing the right level of abstraction for information granules in specific situations. In conclusion, we think a systematic approach to GrC and SA is mandatory due to the importance that both these research areas give to human-centric information analysis and perception-based reasoning. We think that a bivalent approach that considers both the above-mentioned perspectives is useful for this purpose.