Detection and multi-class classification of falling in elderly people by deep belief network algorithms

Abstract

According to the reports on aging population, the number of elderly people without a caregiver has increased. These people are always at high risks of adverse incidents such as increased blood pressure, a variety of stroke-leading health issues, as well as other accidents resulting in body instability and eventually hazardous falls. An uncontrolled fall can result in far worse situations than the original cause itself, if the unattended patient is not promptly transmitted to a treatment center. To reduce the adverse consequences of such unfortunate events, the demand for intelligent systems to prevent, detect, and report the incidents has significantly increased during the past decade. So far, many studies have been proposed considering different aspects of the fall detection problem, from simple applied systems to complex ones regarding the detection algorithm and feature extraction methods. In this paper, a framework for smart detection, identification and notification of elderly falls is introduced. Using a personal smartphone, the tri-axial acceleration of the person’s movements is measured, and the related features are extracted following a pre-processing and timing the samples with a predefined window. The deep belief network (DBN) is used next for training and testing the system using two public datasets, with nine classes of fall and one class of daily activity. Simulation results on two generic datasets, TFall and MobiFall, show an accuracy of 97.56% sensitivity and 97.03% specificity, which is promising compared to nine other related studies.

Appendix: Detailed literature review

In this Appendix, a detailed review of the previous studies is presented. This review is based on the categorization introduced in Sect. 2 (also shown in Fig. 2): type of sensor, sensor placement, type of fall, pre-processing, features and classification algorithm. In the following sections, these categories are described.

1.1 Type of sensors

Different types of sensors are used to recognize physical movements and especially for fall detection, including:

• Accelerometer: a well-known sensor acquiring movement acceleration in three dimensions of x, y and z. The sampling frequency of this sensor is usually between 20 and 100 Hz on average, and the maximum range is $\pm 16$ GHz generally.

• Gyroscope: acquires angular velocity, with a maximum range of 2000 $^\circ /s$. The number of smart phones with a gyroscope sensor is much lower than those with an accelerometer (36% vs 93%) (Phone Finder Comparnion 2018). In addition, according to Figueiredo et al. (2016), the energy consumption of gyroscope sensors is more than that of accelerometers, hence its lower use.

• Magnetometer: used in some previous works including Figueiredo et al. (2016), Ozdemir (2016) and Nguyen et al. (2018), measures direction, strength and changes in magnetic fields. This type of sensor is also of less use in research.

1.2 Sensor placement

Various research has been performed on the sensor location achieve the best accuracy. Since, the sensor must always be in close proximity of the subject, their comfort is also increasingly becoming important beside accuracy. Body parts that are involved in daily activities are not suitable for sensor placement. Sensors located in places such as wrist, ankle or around the neck increase the probability of false alarms due to daily activities.

More accuracy can be obtained by increasing the sensor proximity to the body’s center of gravity, somewhere above the waist and below the neck (Tong et al. 2013). As placing devices such as smart phones in such locations can be unpleasant, a nearby place such as some area below the waist and above the thigh can be useful with acceptable accurately using efficient algorithms (Preece et al. 2009). As such, smartphones inside pants pocket or waist bag are more acceptable with reasonable accuracy than any other areas of the body. In this study, smartphones are considered in frontal pants pocket.

1.3 Fall type detection

While detecting the fall event itself is the main goal in some studies, e.g. (Igual et al. 2015; Abdali-Mohammadi et al. 2016), some others consider using various algorithms for classifying the type of fall. Authors in Dai et al. (2010) among other articles classify falls in three types: forward fall, backward fall and lateral fall. The work in Jian and Chen (2015) detects backward falls. In Ando et al. (2016), several daily activities in addition to fall are also detected, including: walking, sitting, running, going upstairs/downstairs, and laying.

In this article, nine types of falls are considered (according to Table 5, Tfall dataset), including: sideward-lying, back-sitting-lying, forward, forward straight, backward, lateral left and right, sitting on empty air, syncope and forward fall with obstacle.

1.4 Data pre-processing

The data generated by sensors may include noise caused by environmental factors. To reduce the effect of noise, one of the following pre-processing methods can be used:

• Average filter: as a time-domain filter, it is considered the most common filter in signal processing (Khan 2016; Khan et al. 2011). Usually two-point moving average filter is chosen to remove possible noise in movement signals.

• Low pass filter: This filter, which passes through frequencies less than a cut-off frequency, is used to reduce noise and isolate the signal (Smith 1999; Figueiredo et al. 2016).

Generally for detecting falls, using a simple filter is more desirable, to reduce overhead and increase processing speed. Hence, in this article a four-point average filter is used.

1.5 Features

Extracted data from various sensors are not beneficial in raw state and cannot be categorized. Therefore, meaningful features are extracted that accurately represent the original data. There is a large body of research on extracting the motion sensor information, divided into three categories:

• 1. Time-domain

• 2. Frequency-domain

• 3. Discrete wavelet transform

Time domain features, such as standard deviation and average, are used to detect fall from daily activities. Frequency domain features, such as energy and activity entropy, are used to identify the type of fall. For example, in Abbate et al. (2012), some features based on different threshold in time domain or statistical methods were created using a set of conditions. After receiving the data the authors used three time parameters to create their features: peak time, end of impact and start of impact. Next, they generated features such as impact duration index, maximum peak, free fall and similar, and determined a threshold value for every feature, to distinguish falls from regular activities. These times cannot vary from one person to another, and are obtained by analyzing a particular dataset. Therefore, these features will not be accurate for data outside of the dataset being used.

In Kau and Chen (2015), the whole process of fall detection is performed on a person’s smartphone, with no data segmentation for reducing the computation size. The authors only calculate two features from the data: acceleration magnitude and discrimination between similar movements of fall (such as going down the stairs, jumping, getting in a car etc.). Since the sampling frequency is higher than 150 Hz, analysis of input data takes excessive time and energy.

In Shibuya et al. (2015), simple features such as difference between minimum and maximum of acceleration and gyroscope values are used on three x, y and z axes for each time window. One of the reasons for using simple features with low complexity is for use in energy constrained smart phones.

In Aguiar et al. (2014), authors use a number of statistical features to solve the fall detection problem. These statistical features are calculated on data obtained from the x and z axes. These features include average, minimum, maximum, standard deviation, entropy, etc. The entropy feature used in this study is relevant due to the various irregularities observed in daily activities and falls. In addition, in Yang et al. (2016), correlation between axis pairs x–y, y–z, and x–z were also proposed.

Feature extraction in Tong et al. (2013) is performed in such a way that a certain number of samples occur every 10 ms, namely TS. To reduce the dimensions instead of using individual components, sum of squared acceleration components are used to obtain the necessary features for fall detection.

In Micucci et al. (2015), two time windows are used for fall detection. First one is 2.56 s, containing 128 samples, and the second one containing 51 samples. Features addressed in this article include four categories for both types of time windows: raw data, magnitude, statistical features of acceleration (including mean, standard deviation and energy) and local temporally template (created by using the comparison of one sample with its neighbors, with the number of neighbors being six). While this set of features leads to proper results, it requires preprocessing and causes high energy consumptions making it inappropriate for a real-time smartphones application.

In Preece et al. (2009), a comprehensive comparison is presented on a variety of methods for extracting features from accelerometer data. In this comparison, the performance of discrete wavelet transform (DWT) is compared to other feature extraction techniques, related to time and frequency domain features. It is shown that the DWT doesn’t gain adequate accuracy results against time and frequency domain features. As in other work such as Kau and Chen (2015), Shibuya et al. (2015), Aguiar et al. (2014) and Yang et al. (2016), better results are achieved using time and frequency domain features.

1.6 Fall detection algorithms

The algorithms used to detect falls are divided to two categories (Pannurat et al. 2014):

• 1. Threshold-based (TB) or decision making

• 2. Machine learning based

In threshold-based (or decision making) algorithms, the data obtained from the sensors are analyzed and some conditions are determined for detecting fall. For example, the acceleration magnitude is usually the most important component, with a threshold value defined for it.

In Bai et al. (2012), the authors have investigated fall event by using a five-state threshold based algorithm. First, the weightless state is examined and if it is detected, comparison of the acceleration magnitude with a threshold value is done in the second step. If the acceleration magnitude is larger than the threshold, it is investigated whether the impact has happened once more in 2 s. If the event is repeated after 2 s, the activity is classified as running, otherwise a fall event is detected.

In Huynh et al. (2013), the authors use gyroscope and accelerometer sensors, with two thresholds as upper and lower values. These threshold-based algorithms have low computational complexity, which is the reason for their use in smartphones. On the other hand, no comprehensive data are available to find a suitable boundary under different circumstances. Therefore, decision on setting a strict and deterministic value as a threshold increases false positives and false negatives in some cases (Igual et al. 2013). Moreover, laboratory data are usually acquired from daily activities and experimental falls by young people under 30. These data are different in the pattern from those of elderly people, thus, the specified boundaries are not realistic.

Machine learning algorithms are usually divided into three main categories:

In supervised learning, the goal is to learn the mapping between input and output, according to existing labels. Data in this approach are in the form of (inputs, labels) pairs, the set of which is a training set. Many of the algorithms used for detecting fall are based on this approach. For instance, the K-Nearest Neighbors (KNN) algorithm, a non-parametric machine learning algorithm, acts as if a movement instance receives the appropriate label with respect to most labels in its nearest neighbors. The distance from each sample is measured relative to other samples with different criteria, the most common of which is the Euclidean distance. Other criteria include the Mahalanubis, Hamming, Manhattan and the KNN metrics (Mulak and Talhar 2015). This algorithm has a time complexity of O(nd + kn) (with n as the number of samples, d as the number of features and k as the number of neighbors).

Researchers in Jian and Chen (2015) welcome this algorithm to detect falls using smartphones, applying the Euclidean distance criterion. In Gibson et al. (2016), the authors use the KNN as one of the algorithms in a comparator system that includes a set of classification algorithms. Determining whether or not a fall has occurred in this system is performed by determining the majority in various classification algorithms. Although the value of k should not be very high or very low, a standard range this value is not determined, and the most suitable k value is empirically obtained proportional to the number of samples. In this research, k equals 5.

Another algorithm proposed in Medrano et al. (2014) is to use a combination of the K-means and KNN clustering algorithms, such that clustering is performed before the input data are grouped using KNN. A value of k equal to 800 results in the most precise values compared to using KNN alone, which is a very time consuming process.

In Zhang et al. (2006), researchers identify the fall of the single-class SVM (OSVM) from daily routine movements. In this study, falling data are identified as positive data and other movements are considered as anomalies. This way, only the fall data in the training phase are categorized. In the testing phase, the identified data are considered as fall only.

In Shibuya et al. (2015), authors implement a real-time system using the SVM algorithm with a Radial Basis Function (RBF) kernel. In addition to daily activities studied in other studies, movements such as rotation while walking and impact on the obstacle are also considered. In addition to the use of simple features such as the difference between maximum acceleration and its minimum, the maximum angular velocity and its minimum on all three axes, x, y and z, result in acceptable results.

In Micucci et al. (2015) and Yang et al. (2016), a comprehensive fall detection study with two-class classification is conducted, where a fall is considered as an abnormal activity in daily movements. Considering the fact that obtaining data related to falling is one of the main problems, it is suggested to use single-class classifier trained with daily routine data. Therefore, all data detected as abnormal activities are considered as falls. Although not requiring fall data samples, this method can lead to many false alarms. In this study, both the SVM and KNN classifiers are trained and evaluated for both single-class and two-class classification. For the SVM classifier, the RBF kernel succeeds for non-linear classification. In this comparison, performed between four classifiers (single-class and two-classes of KNN and SVM), the best result is obtained using SVM with two-classes according to different criteria.

In Diep et al. (2013), researchers use acceleration data from wearable sensors with a simple feature extraction method. The features are extracted from slices of a sliding window set on the sensor data streams. They applied the SVM classifier for a two-class classification on a dataset with 144 samples for fall as well as other daily activities.

In Abbate et al. (2012), the authors use artificial neural networks for fall detection, specifically a multi-layered nerve grid. By selecting the appropriate number of neurons in the hidden layer, seven neurons are selected as the most suitable number of layers. The Sigmoid function is used as the activator function. Appropriate results are obtained by comparing three threshold-based methods in identifying the fall error. Considering the complexity of the neural network, there has been no evaluation of the use of energy consumption for applying this method on smart phones.

Authors in Özdemir and Barshan (2014) address the fall detection problem using neural networks beside some other classifiers. The neural network used has three hidden layers and 30 neurons per layer, and the Sigmoid activation function is used for hidden layers. For the last layer, a linear function with a threshold of 0.5 is used, detecting a value greater than 0.5 as daily activity and less than that as fall. Also, in terms of the classification speed, neural network training takes more time over other classification methods, such as SVM, KNN, and Least Square Mean (LSM). In fact, the training phase spends most of the time and the time for testing phase is lower.

Another type of neural network used for fall detection is the Multi-Level min–max Fuzzy neural network (MLF) in Davtalab et al. (2014), as a variation of the developed Fuzzy Min–Max neural network (FMM). In general, the fuzzy min–max network, used for non-linear classification, applies hyper-boxes for sample types. Hyper-boxes are identified with two min and max points, which are generated by the arrival of training samples at the time of network training. Hyper-boxes for different classes cannot overlap. Depending on the number of hyper-boxes (or in fact, detection levels), accuracy and network training time rises. Finding other parameters such as the maximum size of the boxes and their shrinking factor to find the most accurate category requires a lot of time while training the network. In Jahanjoo et al. (2017), using this network, the authors were able to identify the fall with better results than other three methods. Since the time for finding the best parameters is a critical factor in this method, they suggested the use of meta-heuristic algorithms.

In unsupervised learning, the data output is given, but there is no indication that the input data are generated by these outputs (Murphy 2012). In fact, there is no tagging that represents the mapping between inputs and outputs. The goal is to learn the structure and the model that each of the data categories has created for a particular output. This approach is appropriate for identifying models whose data labelling is difficult and costly, or all data types of a model do not exist for network training.

An example of this type of learning is the Hidden Markov Model (HMM). Due to its good performance in audio, handwriting and other fields, the authors looked at the sensor data as time series in Tong et al. (2013) and used it to identify falls. In this study, features of the motion time series are generated for each window, and the hidden states are the states that the system can detect. For training, a model is created that will generate the possibility of creating a hidden state according to the observed state. To train the model parameters, the Roof–Welch algorithm was used, which is a specific model of the general Expectation Maximization (EM) algorithm. This algorithm estimates the maximum probability and the posterior state for HMM parameters. In this study, there were three hidden states: balance, loss of balance, and imbalance. Finally, to detect fall, there was a need for a threshold on the likelihood of falling. Therefore, by using the SVM algorithm, the authors separated the threshold probability of daily activity samples with the lowest probability threshold for fall samples. This algorithm was performed on data gathered from eight people, all of them under the age of 28. The data samples used were from 80 fall situations in total for training and testing, and optimal result was achieved for this set of data.

In semi-supervised learning, a combination of the two previous types, both labeled and unlabeled data are used. One type of these algorithms is that labeled data are first used for training. Next, unlabeled samples are gradually introduced into the training phase, and these steps are consecutively repeated until a proper cluster is reached (Chapelle et al. 2010). This type of learning has been considered less in fall detection.

An example of this learning type of is presented in Fahmi et al. (2012), which used the decision tree algorithm to identify the fall situation from accelerometer and orientation sensors. After certain thresholds are obtained, determining the start time of weightlessness, is equal to the start of the stroke or motionless. After falling, changes in angle ​​and acceleration values are calculated. Finally, for different situations, distinguishing falls from other daily activities is performed using the threshold-based constraints derived from the decision tree.

Finally, deep learning methods have also been applied in recent years as in Fakhrulddin et al. (2017), Musci et al. (2018), Torti et al. (2018) and Pena Queralta et al. (2019). In Fakhrulddin et al. (2017), the convolutional neural network (CNN) is applied with data input mapped to grayscale images, representing the features. In Musci et al. (2018 and Torti et al. (2018), which are similar studies from the same authors, the LSTM network is used as one method on Recurrent Neural Networks (RNN) for the fall detection problem. The study in Pena Queralta et al. (2019) similarly uses RNN (LSTM) for this problem, with the difference that in Musci et al. (2018) and Torti et al. (2018) data from two types of sensors are used (accelerometer and gyroscope) while in Pena Queralta et al. (2019) three types (accelerometer, gyroscope and magnetometer) are used. In general, deep learning methods yield more accurate results despite their higher computation time in the training phase. Our proposed method is based on deep learning and different types of fall are detected using a variety of different features with high accuracy. Similar to some of other studies, general datasets are used in this study so that comparison is also feasible. Different from others, the deep network used in this study enables accurate detection of different types of fall.