1 Introduction

Ambient intelligence (AmI) is a novel design paradigm in artificial intelligence (AI) (Ramos 2008) that introduces more intelligent interaction, which, in turn, provides better support for its human users and access to essential knowledge in order to make better decisions. Thus, users become the center of an intelligent digital environment whose primary goal consists of satisfying users’ requirements with regard to controlling the conditions of their surroundings (Sangil 2016). The applications that transform sensor data into a decision are called “context awareness” and are usually borrowed from the well-established AI field. This context awareness is recognized as an important feature of AmI, where a multitude of smart devices and sensors gather data, interact, exchange information, and provide various significant services in our lives (Ramos et al. 2008). Context awareness is becoming increasingly more prevalent and can be found in the areas of wearable computing, robotics, mobile computing, smart interfaces, adaptive computing, augmented reality and intelligent environments.

Through context awareness, one of the main applications of AmI is responding to the presence and behavior of individuals and assisting them anytime and anywhere in their everyday activities by providing an intelligent environment. This requires the accurate detection of the user’s situation, prediction of future events, and real-time decision support with intelligent analysis of large amounts of context data gathered from many sensing devices. Thanks to advances in the miniaturization of sensors and powering IoT devices, it is becoming possible to provide such intelligent developments in AmI (Ahlgren et al. 2016; Maity and Park 2016). In order to provide such services, AmI must be able to learn about its environment over time and react rapidly upon encountering a new situation that arises due to the dynamic change of context. Though prior approaches have presented solutions for context awareness for different domains, they did not satisfy the requirements of AmI. Today, the development of context awareness in the AmI environment is still a complex and time-consuming task due to the lack of appropriate applications architecture support.

This requires a scalable system with high processing capabilities to combine multiple contexts of multiple sensors, each of which possesses individualized context awareness services (Ogie 2016). However, there are related challenges that are currently in open research that first need to be overcome in order to realize the true essence of an AmI system. The first issue is that each sensor has its own unique technical structure and that different processing techniques are needed to sort each sensor’s data (context) and extract useful information first. The second issue is that even if the multi-context data is sorted, it can be challenging to find an intelligent way to combine this information stemming from multiple contexts. Though several approaches have been developed (such as Kalman filters, co-training, and ensemble learning) to combine multiple knowledge contexts, these have not been designed for AmI tasks, nor are they optimal approaches.

This paper seeks to address these issues and develops a context awareness framework called AC (awareness-cognition) for AmI that also solves problems with predictions by discovering personalized knowledge through combining multiple contexts. This research focuses on a middleware layer of an AmI system that handles the context-aware analyses and exhibits high accuracy and efficiency. The main challenges in the work are (1) a multidimensional cube that provides a knowledge structure and combines multiple contexts, (2) a fusion behavior mechanism that can understand activity context information in context-aware environments. I am proposing a general and modular framework to develop a methodology and architecture for the implementation of AmI middleware while considering AI methods and techniques. It is important to note that the AC framework provides a format definition of the context awareness middleware problem. Also, this is centered on the concepts of and oriented for intelligent behavior, in spite of also being oriented for technologies.

2 Principles of context-aware middleware

A variety of context awareness solutions exist on AmI system where the research focus is restricted to specific services such as extracting context information, context reasoning (Jianping and Yu 2010), activity monitoring (Kozina et al. 2011; Forkan et al. 2015), and different application settings (Gravier et al. 2015; Sain et al. 2010). Most of these systems have proposed different context modeling approaches depending on the problem domains. No precise standard is found in the literature for extracting and modeling context information from multiple sensor data especially in the AmI middleware layer. There is no awareness middleware that covers all types of application settings, provides adaptation toward changing context, and acquires a collection of and solutions for user behavior data. Some studies (Wang et al. 2009; Jianping et al. 2010; Sain et al. 2010) have proposed context awareness middleware that provides adaptive behavior in the continually changing environment. However, (Lee et al. 2011) argued in “Survey and trend analysis of context-aware systems” that context awareness middleware proposals tend to lack several key capabilities such as semantic interoperability, fault tolerance, distributed data computation, and accurate reasoning. In addition to technological innovations such as artificial intelligence, cloud computing, and big data analytics, the evolution of context awareness middleware should also be noted.

Recently, Xin et al. (2015) presented a review of prominent solutions to context awareness middleware developed from 2009 to 2015. These context awareness middleware frameworks were analyzed and introduced in depth. According to comparative results in that report, there is no context awareness middleware architecture that is suitable for the “AmI environment”. AmI middleware should consist of various other computational tasks such as context modeling, context retrieval and monitoring, information processing and pattern recognition, application behavior, and there are no architectures to satisfy all of them.

The design of the architecture of context awareness middleware begins with the acquisition layer of various kinds, followed by the formalization and inference processing layer, and finally ends with the distribution of useful context to the corresponding applications layer. This means that context awareness is becoming an essential function of software systems, and the complexity of this system is steadily increasing. In order to overcome the research challenges, the following research questions are defined with the aim of achieving a scalable, cognition- and context aware middleware for AmI.

RQ-1 How to process, model and manage multiple contexts of information from multiple devices to retrieve useful knowledge?

This question addresses the scalability issues with high processing capability to combine multiple contexts by developing a distributed middleware. The significance of solving this issue is to use a common framework to ensure that context awareness middleware can support multiple contexts in parallel and can work on large context data gathered from many sensing devices effectively and accurately.

RQ-2 How to process multiple contexts? How to understand and detect behavioral changes in a user/environment by discovering user-behavior knowledge and context fusion correlations?

For the second research question, we have utilized cognition techniques to develop fusion contextual learning models for behavior knowledge discovery. We then use that knowledge to detect future changes. The interrelations between various multiple contexts are exploited for the development of these models. Collecting the past and present situations context provide cues for immediate decisions (da Rosa et al. 2015) and also predicts future behavior (da Rosa et al. 2016). In this solution, changes are detected accurately and can be supported by user-behavior context awareness applications.

2.1 Context modeling

The context modeling layer is responsible for representing and organizing contexts. This is one of the greatest challenges of context awareness middleware. The raw data fed into the context acquisition layer is obtained from various heterogeneous context sources. Each source has its own expressions, so it cannot be employed directly by a system. Therefore, an approach is needed that can define and organize context in order for the system to understand. Due to the diversity of context sources, it is necessary to provide a consistent and extensible context model for other components of context awareness middleware to access and manipulate contexts. This is the basic design question on how to optimally design the model for the computing system. It is very important to find out the answers to the questions, such as to provide computational support for describing, constructing, analyzing and implementing interactions and coordination patterns between contexts.

The description for each context modelling technique is quite detailed. 10 modelling techniques including “Key-value”, “Markup”, “Graphical”, “Object-oriented”, “Logic-based”, “Domain-focused”, “User-centric”, “Ontology-based” and “Chemistry-inspired” will be summarized to give an overview of the most common techniques. The fundamental scheme to examine the available context modeling techniques is based on the data structure used for representation.

Owing to the diversity of context sources, providing a uniform and extensible context model is necessary to enable the other components of our middleware to access and handle context.

2.2 Reasoning context

The context reasoning layer is one of the most challenging ideas and one of the most challenging modern research challenges in that it attempts to use context data and react to contexts intelligently. The context reasoning is a cognition process and acts as a brain designed to discover new facts about context from existing context features and/or reasons with high dimensional context abstraction.

3 Awareness-cognition framework

The main research objective of this paper is to develop an extensible context awareness middleware for AmI that solves the problems of delivering accurate detections and predictions using multiple context data by discovering user-behavior knowledge using context fusion correlations. The AC framework architecture designs a hierarchical structure which is flexible and scalable for knowledge representation in order to deliver appropriate prediction context to the applications that require this data. The main research is presented in two categories: awareness layer and cognition layer, which are the software modules as shown in Fig. 1.

Fig. 1
figure 1

Architecture of AC framework

Full size image

Figure 1 shows the overall process of the AC framework. In relation to the first research question, the Awareness layer is developed utilizing a Multidimensional cube that is capable of processing multiple heterogeneous data sets from a large number of sensors. Moreover, a unified context model is needed to aggregate data from different sensors of an AmI system into a single meaningful context. To address the second research question, fusion contextual learning models have developed cognition techniques. The cognition layer generates a behavior list of awareness information that fit the user’s current needs. The cognition engine works to acquire and provide complete knowledge in order to satisfy different behavior profiles based on an individual’s specialized needs. The cognitive reasoning engine provides a better understanding of the changes in the current environment based on all the available multiple context information.

3.1 Awareness layer

The AmI system also contains many ambient sensors which report different environmental conditions and activities of the environment. All the low-level sensor data must be converted to high-level contexts within the awareness architecture. In the acquisition stage, a great variety and volume of context data composed of multiple formats is obtained. To use them, it is necessary to define and store them in a machine-readable and -manageable format and all data must be converted to a unified format so that the context can be understood and shared. This can be accomplished by the awareness module that defines, expresses, and processes objects. Ontological models are human–machine readable; they can be quite easily understood by humans and are directly interpreted by machines.

By recognizing raw sensing data, the awareness module extraction allows users to better understand the context characteristics. The accurate and clear formatting and segmentation of sensory data are useful for identifying, understanding, manipulating, and sorting out a variety of contexts in an efficient way. The awareness module contains an awareness rule to manage context-life-cycle for context awareness. This rule is included in two significant events as pre-processing and post-processing.

Pre-processing explains how context is extracted and formalized, useful in the inference procedure, which requires some level of data abstraction to retrieve the extracted context values. To represent a domain, its information, and its context space, we have the following two definitions.

Definition 1

Context c is a variable that characterizes the situation of the user. The set C of n context is defined as:

$${\text{C}}={c_1},~ \ldots ,~{c_n}$$

Definition 2

Let \({\text{c}} \in {\text{C}}.\) The values that correspond to the context \({\text{C}}\) are called context value. The set that contains all the possible values of the context \({\text{C~}}\) is defined as \({V^c}\). The context values can be numerical or categorical and are marked with \({V^c}\), where \({v^c} \in {\text{~}}{V^C}\).

Definition 3

Perception vector \({x_t}\) is a multi-dimensional signal vector containing values from each sensor associated to a time point \(t\). It is assumed that it is possible to build perceptions from all sensors, regardless of how often a particular sensor provides measurements.

Definition 4

Perception sequence \({X_{t,T}}\) with length \(T\), is defined as a vector of perception vectors, \({X_{t,T}}=\{ {x_t},~{x_{t+1}},~ \ldots ,~{x_{t+T}}\}\). The observation sequence consists of the measurements that correspond to the particular time interval (window), \({w_{t,T}}\).

Definition 5

Let \({\text{X}}\) be a set of perception sequences, \({X_{t,T}} \in X\). Let \({V^c}\) be a set on context values for particular context c. Perception across is a process of applying a function \({f^c}\), which takes as input an observation sequence \({X_{t,T}}\) and gives as output a context value \({v^c}\), where \({v^c} \in {\text{~}}{V^C}\):

$${f^c}:X \to {V^C}$$

The post-processing is a process of constructing a multidimensional cube which uses said contexts to make a decision. It exploits each of the extracted contexts individually, and the data from the other contexts are used as reasoning data. In other words, context-based data partitioning is performed. The context values segment the whole data set into subsets for which a context model is constructed. Therefore, the awareness of a particular context value \({v^c}\) of a particular context c can be defined as a function, which transforms the multidimensional cube into a decision. Thus, the following three terms are defined: reasoning multidimensional context, context-based data partitioning, and multidimensional cube modeling:

Definition 6

Reasoning \(A{R_{{V^C}}}\) is defined as a subset of the whole dataset, which contains only the data instances that have that particular value \({v^c}\) for the context c. Partitioning is the process of creation of the aware reasoning \(A{R_{{V^C}}}\) for the context value \({v^c}\).

Definition 7

Multidimensional cube modeling \(md_{v}^{c}\) is defined over the aware reasoning data \(A{R_{{V^C}}}\) for the context value \({v^c}\). The set of all context models (created for all context values for all context) is defined as \({\text{MD}}\) and \({m^c} \in MD\). The size of the set \({\text{MD}}\) is defined as \({\text{Z}}\), where \(\left| {{V^c}} \right|\) is the size of the set of values for the context c:

$${\text{Z}}={\text{~}}\mathop \sum \limits_{c}^{n} \left| {{V^c}} \right|$$

The pseudo code of the pre-processing phase in awareness layer based on definition (2–5) is given with Algorithm 1 and pseudo code of the post-processing phase based on definition (6, 7) is given with Algorithm 2:

figure a

In the first phase (A) the raw sensor data are acquired and the multiple contexts are extracted using different types of techniques: pre-processing, synchronization and data segmentation. In our approach, the context represents is information about the user which was extracted from the sensor data. In phase B, the aware reasoning is performed using the contexts defined in the previous phase. First, the partitioning of the dataset is according to each context and its values. Therefore, for each context value a multidimensional cube \(m{d^c}\) is constructed using its reasoning data—the reasoning data is a subset of the whole dataset that has that particular context value \({\text{~}}A{R_{{v^c}}}\). For example, the reasoning data for the “HOME” model will be constructed using the data instances that contain the value “HOME” for the activity context. This way, the approach considers multiple views on the data using each of the features as a context.

3.2 Cognition layer

The cognition module generates an action list of cognitive information that meets the user’s current needs based on an understanding of the flexible and common integration of cognitive categories. Cognition techniques were used to develop fusion contextual learning models for behavior knowledge discovery which, in turn, are used to detect future changes. The operating layer of the cognitive layer module is accompanied by a knowledge fusion process involving the fusion of information and knowledge in the existing structure of the multidimensional context. In addition to this, multiple operating contexts allow the fusion of knowledge from different contexts and the discovery of new relationships between the instances involved in these contexts. This cognition layer module consists of two parts: the fusion and behavior mapping process—see Fig. 2.

Fig. 2
figure 2

The structure of cognition layer

Full size image

The fusion context defines the relationship between entities in the multidimensional cube based on a Snowflake schema. A snowflake schema is a logical arrangement of the table in a multidimensional cube such as in an entity relationship diagram. The dimensional relationship is represented by centralized fact tables which are connected in multiple dimensions, and also dimensions are normalized into multiple related tables. When dimension tables are connected, you can define a single dimension using columns from multiple tables, or you can define separate dimensions based on the separate dimension tables and then define a link between them using the reference dimension relationship setting.

The fusion context can lose information about the interrelationships within the context. Context modeling expands the dimension of the problem in order and fosters multiple relationship aspects to this problem. Therefore, it needs to provide more accurate relationships and should give meaning to user and context. The main goal of behavior mapping is given some prior information about users and multidimensional context, trying to understand what particular context will be the most relevant to a selected user. Following the MDM (Multidimensional Dataspace Model), in which (Adamovicius et al. 2005) proposed a reduction-based approach, we can reduce the problem of multidimensional (MD) contextual recommendations to the standard 2D \(R_{{USER~ \times ITEM}}^{D}:U \times I \to Rating\) recommendation space. In MDM, the dataspace is the Cartesian product of \({N_D}\) dimensional: \({\text{DS}}={D_1} \times {D_2} \times \ldots \times ~{D_{{N_D}}}\).

Each dimension contains one or more attributes: \({D_i}={A_{i,1}} \times {A_{i,2}} \times \ldots ~ \times ~{A_{i,{N_i}}}\). The data model is very similar to that of a relational database. It usually also requires that the values of an attribute come from a set of atomic and nominal attributes. The definition of behavior mapping is as follows:

$$FM_{{Person~ \times context}}^{D}:{\exists _{{v^i}}} \cdot {P_{id}} \times \left[ {{\forall _{{v_i}}} \cdot {D_i}} \right] \sqcup \ldots ~ \sqcup \left[ {{\forall _{{v_i}}} \cdot {D_n}} \right]~$$

Where FM is a behavior mapping modeling that extracts information about a specific user. The \({\exists _{{v^i}}} \cdot {P_{id}}\) are some person’s values from dimensional \({P_{id}}\) by combining each relation contexts in the other dimensional \(\left[ {{\forall _{{v_i}}} \cdot {D_i}} \right] \sqcup \ldots ~ \sqcup \left[ {{\forall _{{v_i}}} \cdot {D_n}} \right]\) to extract semantic information (Fig. 3).

Fig. 3
figure 3

Behavior mapping

Full size image

The CF(303, I302,2,5, ASP1) = 7 in the Fig. 3, is mean that for the person with ID 303, the user’s action is was ID I302 and using this action mainly time ID5 in the place company, specified during in the device ID ASP1, saving that information ID 7. In the other words, the person in afternoon is using application (ID I302) at the company using a smartphone. That contextual information is basic for a creative user behavior historical database.

3.3 Integration and discussion

Context aware middleware architectures are analyzed based on several parameters including fault tolerance, adaptability, and scalability. In this paper, we will logical compare and evaluate the proposed context middleware architecture from 2012 to 2015. It is feasible to analyze the performance of context aware middleware architectures from various points of view. The middleware proposed in this paper considers the following important attribute list.

3.3.1 Interoperability

The definition of interoperability is “the ability of two or more systems or components to exchange information and to use the information that has been exchanged” [IEEE Standard Computer Dictionary].

3.3.2 Context reasoning

Context reasoning can be defined as a method of deducing new knowledge, and understanding better, based on the available context. Reasoning performance can be measured using efficiency, soundness, completeness, and interoperability.

3.3.3 Context pre-processing

This phase cleans the collected sensor data. Due to inefficiencies in sensor hardware and network communication, collected data may be not accurate or missing. Therefore, data needs to be cleaned by filling missing values, validating context via multiple sources, and many more.

3.3.4 Sensor data fusion

It is a method of combining sensor data from multiple sensors to produce more accurate, more complete, and more dependable information that could not be achieved through a single sensor.

3.3.5 Knowledge context

Most of the tasks that are performed by AmI middleware solutions require knowledge in different perspectives, such as knowledge on sensors, domains, users, activities, and much more.

In quantitative analysis, our middleware performs well with respect to architecture and supporting context reasoning. With respect to multidimensional cube, cognition fusion on multiple contexts, our middleware performs better than others middleware systems. Many other aspects are involved and need to be addressed in terms of real development, such as problems with the privacy protection and Fault Tolerance that are not considered in our middleware. We will try to tackle this problem in a future study. Through continuous improvement, our goal is to provide developers with a better platform for developing context awareness applications in ambient intelligence environments.

4 Conclusion

This work described the design of a comprehensive architecture of knowledge management in the middleware of Ambient Intelligence. We have introduced a context awareness framework called awareness-cognition (AC) that focuses on the middleware layer of the AmI system. This framework has solved the prediction problem by combining multiple contexts to discover personalized knowledge. We have developed a fusion contextual learning model for behavioral knowledge discovery and have used cognition techniques based on knowledge to detect future changes. The main research has provided three categories of awareness and cognition layers.

The Awareness layer was developed using multidimensional cubes and processed multiple heterogeneous data sets from multiple sensors. To approach a single situation, we provided modeling to aggregate data from different sensors in the AmI system into a single meaningful context. The cognition layer has created a list of behaviors of the cognitive information that fits the user’s current needs, based on an understanding of the flexible and common integration of cognitive categories. The cognition engine provides complete knowledge to meet diverse behavioral profiles according to the specific needs of the individual. Based on cognition inference engines, the system can better understand changes in a user’s current environment based on all possible multiple contextual information.

The solutions addressed in this thesis the scalability issues with high processing capabilities to combine multiple contexts and can work on large context data gathered from many sensing devices effectively and accurately. We believe that this research is a needed step towards building a generic model for the AmI system and can be used for the development of efficient learning models for the user in context-aware environments. There are still many problems to be solved. In the future, we plan to do a lot of activities based on this research and develop a part of the AC framework.