Keywords

1 Introduction

The world continues to age rapidly, and the number of persons aged above 65 years is expected to reach 1.5 billion in 2050 [14]. Countries are therefore grappling with the issue of facilitating independent living for the elderly. With labour shortage being another issue, substantial investment is directed toward developing automated monitoring capabilities for the elderly. There are numerous approaches in literature for detecting the daily activities of the elderly (and also in general) in indoor settings through the use of wearable sensors [7], ambient sensors [16], and vision sensors [17, 18].

Anomalies in the daily activities of the elderly may be classified as ‘sudden’, including phenomena like falls, and ‘behavioral’, including variations in the normal routine. Behavioral anomalies are further classified as ‘short-term’ and ‘long-term’ based on the nature of the variations. Short-term anomalies include: variations in performing a single activity; variations in the order of performing a set of activities; and variations in the activities of the entire day. Long-term anomalies, on the other hand, encompass variations in the daily routine of an individual over time. A few examples of long-term anomalies in the elderly are: a person starts spending more/less time in bed; starts using the toilet more often than usual; stops going out; misses important activities often. Some of these anomalies fall in the category of short-term anomalies, but the consequence of such anomalies in the short term is not severe. When these patterns repeat for a longer duration, they have severe consequences and are termed long-term anomalies. A brief taxonomy of anomalies in the elderly is included in Fig. 1.

Fig. 1.
figure 1

Classification of anomalies in elderly care.

Full size image

Approaches in literature are mostly capable of recognizing sudden anomalies like falls [2, 11] using different types of sensors. Few approaches analyze changes in behavior to detect short-term anomalies [1, 3, 8, 9, 12, 15]. The few endeavors in the literature that focus on short-term behavioral anomaly detection take one of the following approaches: activity-wise anomaly detection [8, 12] that detects abnormalities in the execution of an activity, i.e., changes in the duration of the activity, or the location where the activity is performed; and sub-sequence wise anomaly detection [3, 9] that looks at anomalies in the ordering and sequencing of activities over a short period of time, and day-wise anomalies [1, 15] that is also based on ordering and sequencing of activities but over the duration of an entire day. Most of the existing activity-level anomaly detection systems ignore the ordering aspect of the activities; on the other hand, sub-sequence and day-level anomaly detection approaches do not consider the abnormality in the execution of individual activities. Furthermore, they miss out on detecting and comprehending patterns in behavioral change that may not require immediate intervention but can indicate long-term pernicious health issues.

These approaches classify behavioral anomalies either at a fine-grained level and look at individual activities separately or at a more coarse-grained level and focus on the sequence of activities over the whole day. There needs to be, therefore, a robust mechanism that immediately alerts the concerned caregiver for all types of short-term anomalies and also keeps track of long-term anomalies that eventually, over a period of time, indicate a behavioral change and perhaps the build-up of an adverse medical condition. This paper aims at comprehensive anomaly detection over both the short-term and the long term.

2 Anomaly Detection System

Existing approaches for activity monitoring and anomaly detection, as discussed earlier, look at limited portions of the activity network and are unable to study behavioral patterns comprehensively. Our objective is to develop a complete tool for monitoring the behavioral patterns of the elderly over a short and long duration. The proposed system detects anomalies at the level of: 1) individual activities; 2) within sub-sequences of activities; 3) within longer sequences of activities covering the entire day; and 4) finally, within activity networks spanning multiple days. Such comprehensive analysis and anomaly detection at varying levels of granularity lead to a good understanding of the behavioral patterns of the individual. This facilitates the detection of anomalies in behavior and helps draw sound predictions on potentially adverse medical conditions. The systematic workflow of the proposed system is shown in Fig. 2. The system comprises six modules starting from activity detection and extending up to the point that the system informs concerned personnel about the well-being of the monitored individual.

Fig. 2.
figure 2

Workflow of the proposed Comprehensive Anomaly Detection System(CADS).

Full size image

2.1 Activity Recognition and Sudden Anomaly Detection

The activity detection and recognition module detects and classifies activities from streaming data emanating from various sensors (i.e., wearable, ambient, or vision). The proposed framework works at the activity level and can utilize any of the existing activity detection and recognition systems depending on the sensors involved.

2.2 Activity Level Anomaly Detection

Activities are the basic building blocks of the behavioral patterns of the elderly and describe how an elderly person performs his/her daily activities. Let \(A=\{a_1,a_2,..., a_n\}\) be a set of n activities that a person performs on a daily basis. The daily activity routine is defined as a tuple:

$$\begin{aligned} R = \{a_i,S_i(t),D_i(t)\} \end{aligned}$$
(1)

where

  • R: Sequence of activities performed on any given day.

  • \(a_i\): \(i^{th}\) activity of the day.

  • \(S_i(t)\): Start time of the \(i^{th}\) activity.

  • \(D_i(t)\): Duration of the \(i^{th}\) activity.

Each activity \(a_i\) is characterized by two features \(S_i(t)\) and \(D_i(t)\). An activity is classified as normal or anomalous based on significant variations in \(S_i(t)\) and/or \(D_i(t)\). For example, suppose the normal time of having breakfast for an individual is around 10:00 AM, but on a given day, he/she takes breakfast at 12.00 PM; this should be treated as abnormal. Similarly, the usual duration for taking a bath for a person is 15–20 minutes, but on a given day, the person takes 1 h; this should also be treated as abnormal.

The first step of any anomaly detection system is defining what constitutes a normal pattern. For activity level anomaly detection, the normal pattern is defined separately for each activity based on the available historical data over several days. We harness a One-Class Support Vector Machine (OC-SVM) algorithm [13] to model the normal pattern of activities. OC-SVM is an unsupervised learning approach that creates a hyper-sphere for normal patterns. Figure 3 demonstrate a hyper-sphere of radius ‘r’ formed by the OC-SVM algorithm using arbitrary data points. The instances belonging to the normal category are shown by green color points, and the abnormal instances are shown in red. The computation of the hyper-sphere is done using Eq. 2 during training.

$$\begin{aligned} \min _{r,c} r^2 \; \text {subject to,} \, {|| \phi (x_i)-c ||}^2 \le r^2 \; \; \; \; \; \; \forall i \in 1,2,...n \end{aligned}$$
(2)

where r and c are the radius and center of the sphere and \(x_i\) are the training samples. \(\phi \) is the feature transformation function. A testing instance \(x_t\) is classified as anomalous if its distance from the center c is greater than r (i.e., \(x_t\) lies outside the hyper-sphere).

Fig. 3.
figure 3

One Class Support Vector Machine (OC-SVM) hyper-sphere.

Full size image

The anomaly score for an activity instance is calculated based on the distance of the activity instance from the center of the hyper-sphere as described in Eq. 3.

$$\begin{aligned} a\_score_t = \varphi (dist(x_t,c )) \end{aligned}$$
(3)

where dist(*) is a function calculating the distance between a test sample and the hyper-sphere center, and the \(\varphi \) is a function that maps the distance to an anomaly score.

2.3 Day Level Anomaly Detection

Activity level anomaly detection considers the deviations in the execution of an individual activity only without looking at its correlations with other activities. We propose a graph-based approach to model the relationships between several activities. Anomalies in the order of performing activities are then detected using graph-matching algorithms.

The daily activities of an individual are represented using a dynamic graph \(G_{t}\) = \((A_t, E_t, A_s, F, W)\) called the Daily Activity Graph (DAcG). The nodes of the graph denote features describing an activity, and the edges from a node denote the probability of occurrence of subsequent activities. \(G_{t}\) is defined as a tuple, where:

  1. 1.

    \(A_t\) is the set of activities performed till time ‘t’.

  2. 2.

    \(E_t\) is the set of edges between the activities, denoting the order.

  3. 3.

    \(A_s\) is the first activity of the day.

  4. 4.

    F is a feature vector corresponding to each activity, s.t., \(F: A_t \rightarrow R^{d_a}\).

  5. 5.

    W is a weight matrix corresponding to each edge denoting transition probability, s.t., \(W: E_t \rightarrow eVal\).

The DAcG for several days together indicates the ‘normal’ behavior of a monitored individual. We call this \(G_{ref}\) the Reference Graph of daily behavior. A sample \(G_{ref}\) for an elderly person is shown in Fig. 4. The tuple (p, q) on bidirectional edge E(A, B) denotes the transition probability, where p is the probability of ‘B after A’ and q is the probability of ‘A after B’. A probability value ’p=0’ indicates that the elderly person never performs activity ’B after A’.

Fig. 4.
figure 4

Daily Activity Graph for Elderly (Each edge A-B is represented by a tuple (p,q), where p is the transition probability from A to B and q is the transition probability from B to A).

Full size image

A day level anomaly is detected by matching the current DAcG (\(G_t\)) with the Reference Graph (\(G_{ref}\)). Graph matching is performed using two similarity metrics, namely, Jaccard distance and Hausdorff distance. The adjacency matrix of the DAcG is a matrix W of size \(n \times n\), where W[i, j] is the transition probability between activity \(A_i\) and \(A_j\).

The Jaccard distance [6] is the measure of dissimilarity between two sets/vectors and is based on the common elements in the two sets. The Jaccard distance between two adjacency matrices X and Y with ‘n’ nodes is defined in Eq. 4. Jaccard distance is ’0’ for similar matrices and is a good measure of similarity/distance between two adjacency matrices of similar dimensions.

$$\begin{aligned} j\_dist(X,Y) = 1- \frac{\sum _{k=1}^{(n \times n)} \min (X_k,Y_k)}{\sum _{k=1}^{(n \times n)} \max (X_k,Y_k)} \end{aligned}$$
(4)

The Hausdorff distance [4] between two matrices is another useful measure of dissimilarity. This distance is different from the Jaccard distance because it forms a many-to-many mapping between the matrices instead of a one-to-one correspondence. This is also known as the maximum distance between the two. The Hausdorff distance between two adjacency matrices X and Y is defined in Eq. 5.

$$\begin{aligned} h\_dist(X,Y) = \max _{x \in X} \min _{y \in Y}||a-b|| \end{aligned}$$
(5)

Jaccard distance and Hausdorff distance are adept at detecting anomalies that occur due to incorrect ordering and missing activities, respectively. However, these distances are not capable of detecting anomalies due to variations in the execution of individual activities.

An anomaly score is established by combining the distances and anomaly scores of all the individual activities. Equation 6 describes the calculation of the overall anomaly score for a day. A higher anomaly score indicates a greater chance of deviations from the normal behaviour.

$$\begin{aligned} anomaly\_score = w_1*\left[ \frac{\sum _{k=1}^{n} a\_score_k}{n}\right] + w_2* j\_dist(G_{ref},G_t) + w_3* h\_dist(G_{ref},G_t) \end{aligned}$$
(6)

where n is the number of activities in a day, \(G_t\) is the DAcG of the current day, and \(G_{ref}\) is the reference graph of the normal behavior of an individual. \(a\_score_k\) is the anomaly score of the \(k^{th}\) activity of the current day. \(w_1\), \(w_2\), \(w_3\) are the weights assigned to each metric. A day is classified as normal or anomalous based on a threshold on the anomaly score. Both weights and the threshold are established during the training process.

2.4 Sub-sequence Level Anomaly Detection

DAcG is dynamic in nature, where a new activity is added as a new node whenever one is detected. A dynamic graph \(G_{t^{'}}\) at any time \(t^{'}\) is a sub-graph of the Daily Activity Graph \(G_{t}\), where \(t^{'}\) is the time of the day. \(G_{t^{'}}\) comprises activities performed till time \(t^{'}\) (e.g., \(t^{'}\)=1.00 PM can include all activities until lunch).

A sub-sequence level anomaly is detected by matching [5, 10] a sub-graph \(G_{t^{'}}\) with the Reference Graph \(G_{ref}\) with the former being classified as anomalous or normal based on the anomaly scores. The sub-graph \(G_{t^{'}}\) containing ‘k’ vertices is first searched in the reference graph and then compared with the corresponding part of the reference graph.

The anomaly score for the sub-graph is calculated in a manner similar to that described in the last sub-section for day-level anomalies.

2.5 Long Term Behavior Tracking

The long-term behavior tracking module is responsible for tracking changes in the behavior of a monitored elderly person based on a comprehensive assessment of his/her daily activities over a period of time (i.e., weeks, months, quarters, or years). This module involves generating periodic graphs and comparing successive periodic graphs to identify changes in the activity patterns, if any. These changes in behavior are analyzed and potentially indicate a gradual deterioration in the health and well-being of the individual.

As mentioned earlier, the DAcG is defined as a tuple \(G_{t}\) = \((A_t, E_t, A_s, F, W)\). The periodic graph \(G_{period}\) = \((A_p, E_p, A_{sp}, F_p, W_p)\) for a certain number of days is generated by combining the DAcGs of these days. \(G_{period}\) is calculated as described in Eq. 7.

$$\begin{aligned} \begin{aligned} A_p & = \bigcup _{i=1}^{m}A_{t{i}} \\ E_p & = \bigcup _{i=1}^{m}E_{t{i}} \\ A_{sp} & = \min _{i=1}^{m} A_{s{i}} \\ F_p & = \sum _{i=1}^{m} F_i \\ W_p & = \sum _{i=1}^{m} W_i \\ \end{aligned} \end{aligned}$$
(7)

where ‘m’ is the number of days in the period (i.e., \(m=7\) for a weekly graph). \(A_{t_i}\) denotes the set of activities performed on the \(i^{th}\) day. Similarly, \(E_{t_i}\) denotes the sequence of the activities on the \(i^{th}\) day.

The change in behavior in a given period of time is computed by comparing the periodic graphs \(G_{period}\) of two successive periods (i.e., week-1 and week-2). Activity level anomaly scores between the two periodic graphs indicate variations in the execution of different activities. Activity level anomaly scores are calculated as per Eq. 3. The distance matrix between the two periodic graphs indicates the variations in the ordering of activities or the number of missing activities.

2.6 Alarm and Profiling System

The alarm and profiling module primarily works towards establishing a robust system for effective communication of an adverse medical condition to designated caregivers of the monitored individual. The communication may be urgent such that it requires immediate intervention (e.g. anomalies like falls), or it can be routine wherein the caregiver is informed about long-term and gradual behavioral changes in the monitored individual, indicating the possible development of an adverse medical condition. This module enables caregivers and doctors to track the health conditions of the elderly periodically and make recommendations accordingly.

Alerts are sent using existing technologies like automated SMS/calls in case of sudden anomalies and through email in case of short-term and long-term anomalies. Profiles are created based on the results obtained from the Long-Term Behavior Tracking module. The profile includes the changes in individual activities and their ordering.

3 Results and Discussion

The evaluation of the proposed comprehensive anomaly detection system is presented in this section. The details of the dataset, including the set of activities and their attributes, are discussed first. Subsequently, the validation of anomaly detection at various level days is presented.

3.1 Dataset

Most of the existing works on anomaly detection use their own dataset and are not publicly available. The few datasets that are available are very small and do not contain a complete set of daily activities and types of anomalies. Therefore, we created a dataset synthetically containing thirteen daily activities of the elderly. To imitate the real-world behaviour of multiple individuals, the start time and duration of each activity is varied randomly. Our dataset contains daily activity patterns of around 350 normal days and 180 anomalous days. A sample of daily activities for the day is included in Table 1. WakeUp is always the first activity, and its duration is kept at zero. goSleep is always the last activity of the day, and the end time of this activity is the next morning.

Table 1. A sample of daily activity patterns.
Full size table

A random degree of variation is incorporated into the activities on normal days by varying the start time and the end time of the activities. The sequence of some of the activities is also altered, assuming that the elderly can perform some activities in an interchangeable order (e.g., takeMedicine and useToilet, or eatBreakefast and takeBath). In the anomalous patterns, the three most common types of anomalies considered are Long Day(LD), Missing Day (MD), and Swapped Day (SD). A long day means one or more activities took a significantly longer/shorter duration than usual, a missing day means one or more activity was not performed on that day, and swapped day means the order of one or more activities is changed, unlike usual. The dataset contains patterns of around 60 d for each type of anomaly.

3.2 Validation of Activity Level Anomaly

The method described in Sect. 2.2 is used to generate the anomaly score for each activity. The daily activity data of normal days is used to train the OC-SVM classifier. Two features, start time and duration corresponding to each activity, are transformed to a similar scale using Min-Max normalization. Before normalization, the start times are converted into numbers. The learned OC-SVM model is then utilized to generate the anomaly scores of the activities of both normal and anomalous days. Each activity is classified as normal or anomalous based on the activity anomaly score. Figure 5 shows the activity anomaly scores of four activities, useToilet, doExercise, eatLunch, and goOut. A small number of days selected from all the days are plotted for better visualization. Figure 5 shows the activity level anomaly score for each activity based on the prediction of OC-SVM classifier trained for the activities of multiple individuals over multiple days.

Fig. 5.
figure 5

Activity anomaly score of four activities: a) Use Toilet; b) Do Exercise; c) Eat Lunch; d) Go Out

Full size image

A higher anomaly score indicates more variations in the activity and is termed an anomaly beyond a threshold. Activity level anomaly occurs mostly due to missing activity or significantly longer/shorter duration of the activity. In Fig. 5(a), the anomaly score of the useToilet activity is abnormal on LD16 and LD46 due to longer duration and an earlier start-time, respectively. Similarly, the bigger spikes in the goOut activity are shown on MD23 and MD38, indicating the abnormality due to the missing goOut activity.

3.3 Validation of Day Level Anomaly

Day-level anomalies are detected based on the overall anomaly score of the given day as described in Eq. 6. The weights and the thresholds to classify a day as normal or anomalous are tuned empirically using the training data. Figure 6 shows the overall anomaly scores of all the normal and abnormal days, and the threshold (‘Th’) separates normal days from the anomalous ones.

Fig. 6.
figure 6

Overall anomaly score of normal and anomalous days.

Full size image

Figure 7 further specifies the types of anomaly that lead to the day being anomalous. Anomaly due to abnormal start time or duration is effectively classified using the average activity anomaly scores as shown in Fig. 7(a). All the anomalous days with longer activities (shown in orange color) are comfortably distinguished from normal days. The H_dist, as given in Eq. 5, is near zero for normal days and has a much larger value for a missing day. Hence, H_dist is an effective indicator of days with missing activities, as shown in Fig. 7(b). J_dist, as per Eq. 4, is high for the days with disordered activities. J_dist comfortably classifies days with unusually swapped activities from normal ones.

Fig. 7.
figure 7

Categorization of anomalies using separate scores: a) Average anomaly scores of all the activities in the day; b) H_Dist; c) J_Dist

Full size image

In the case of an anomalous day, the activity anomaly scores of each activity on the given day also provide insights into the cause of the anomaly. A sample day is shown for each category (normal, long, missing, and swapped) in Fig. 8. Activity anomaly scores of each activity on a normal day are smaller, as shown in Fig. 8(a). In Fig. 8(b), the activity anomaly scores of certain activities are high, indicating abnormality in these activities (i.e., useToilet, eatBreakfast, and watchTV at night). In the case of a missing activity, the activity anomaly score is very high, as shown in Fig. 8(c). A set of activities performed in an incorrect order is indicated in Fig. 8(d) for a day with disordered activities. However, the anomaly score is a comparatively weak indicator of a day with disordered activities.

Fig. 8.
figure 8

Analysis of activity anomaly score of all the activities on a day: a) Normal day; b) Anomalous day where some activities performed abnormally; c) Anomalous day where some activities missed; d) Anomalous day where some activities performed in incorrect order.

Full size image

3.4 Validation of Sub Sequence Level Anomaly

Sub-sequence level anomaly is a subset of a day-level anomaly and has fewer activities. The anomaly scores for sub-sequence level anomalies are calculated in a manner similar to day-level anomaly but using a subset of activities. To validate the effectiveness of the proposed approach in sub-sequence level anomaly detection, we experimented with different subsets of the activities. For example, we considered only the first three (\(k=3\)) activities (i.e., wakeUp, useToilet, and takeMedicine) for any given day. A sub-graph consisting of three activities \(G_k\) is then searched in the reference graph \(G_{ref}\), and the selected part of \(G_{ref}\) is termed as \(G_{ref_k}\). The average activity score, distance, and anomaly scorse are computed for \(G_k\) with respect to \(G_{ref_k}\) in a manner similar to that described in Sect. 3.3.

Fig. 9.
figure 9

Precision of anomaly detection on the sub-sequences of different lengths.

Full size image

Sub-sequences are classified as normal or anomalous based on the anomaly score and other indicators. The precision of the classification for different types of days (i.e., normal, long, missing, swapped) is calculated based on the true class of the sub-sequence. It is important to note that many sub-sequences with the first ‘k’ activities (i.e., \(k=3\)) for an anomalous day can be normal as an anomaly may lie in the upcoming activities. Figure 9 shows the precision of the anomaly detection for the sub-sequences of different lengths. For now, the analysis on missing days is excluded as it is difficult to decide whether the activity is missing or will be performed later in the sequence.

3.5 Results on Long Term Behavior Tracking

Long-term behavior tracking involves the changes in activity patterns over a certain period of time. To simulate the results of long-term behavior changes, both normal and abnormal days are mixed and shuffled. Periodic graphs are created by combining seven normal days, termed normal weeks, and seven anomalous days with similar kinds of anomalies, termed anomalous weeks. Figure 10 shows the behavioral changes in the execution of various activities in four weeks. This contains two normal weeks where the activity patterns are almost similar. A long week was created by combining days with a specific type of anomaly (i.e., doExercise activity with a longer duration), and the spike indicates that the doExercise activity is performed abnormally in the given week. The missing week is created with days having another specific anomaly (i.e., missing the eatLunch activity), and the abrupt change in the plot indicates the missing eatLunch activity in the given week. Such information enables the caregivers to introspect into the issues and take necessary action if and when needed. Similar analysis can be done for any period, such as months, quarters, or years.

Fig. 10.
figure 10

Precision of anomaly detection on the sub-sequences of different lengths.

Full size image

4 Conclusion

In this paper, we proposed a comprehensive system for anomaly detection in the daily activities of the elderly. The proposed system is unique because it not only enables the detection of immediate anomalies but also enables analysis and comprehension of the behavioral patterns of an individual over the long term. An understanding of the latter facilitates early detection and diagnosis of pernicious developments like dementia and Alzheimer’s. We expect the system to be of utility to the elderly and also assist caregivers in providing better care. As the part of future work, we will create a real-world dataset of daily activities of elderly, including adequate number of elderly. Also, adaptability of the framework for different individuals behaviour can be explored in the future work.