Prototype Similarity Learning for Activity Recognition

Human activity recognition (HAR) is a significant step towards human computer interaction and enables a series of promising applications such as assistant living, skills training, health monitoring, and robotics [6]. Existing HAR techniques are either video- or sensor-based. In particular, sensor-based HAR aims at inferring human activities from a set of sensors (e.g., accelerometer, gyroscope, and magnetometer), which generate data streams over time. This approach is generally known to have several advantages over video-based HAR including: ease of deployment, low cost and less invasive from a privacy perspective [7].

Previous studies on sensor-based HAR focus on designing powerful hand-crafted features in time (e.g., mean, variance) and frequency domain (e.g., power spectral density) to represent segments of raw sensory streams [9]. Traditional machine learning models such as Support Vector Machine (SVM) and Random Forest are employed to project the feature vector to activity labels [2]. The performance of these methods normally depends on the effectiveness of the extracted features where are heuristic, task-independent, and not specially designed for HAR [12]. Since designing powerful task-specific features require significant domain knowledge, and are labour intensive and time consuming, recent research introduces deep learning methods, which have exceptional data representation ability to expedite feature extraction. These works utilize deep neural networks, such as Convolution Neural Networks (CNN) [5, 16] and Long-Short Term Memory (LSTM) [8, 11], as feature extractors to learn the representation of the input sensory segments automatically, and then map the representation to labels using another neural network (normally a basic fully-connected layer).

Although deep learning methods have achieved significant progress, it is still difficult to learn accurate representations for the input segments due to the complex spatial correlations among sensors and temporal correlations between time periods. Considering the sensitivity of neural networks to noise, the biases in the representations further prevent neural network-based classifiers from making correct activity classification. In addition, subject variances inherently exist in HAR, where people tend to perform activities that are heavily influenced by personal characteristics, such as gender, height, weight, and strength. For example, men usually perform activities at a larger magnitude than women. Such divergence introduces deviations to the representations among subjects and thus prevent the model from getting accurate classification for new subjects (haven’t appeared in the training set).

We propose to solve this problem from three perspectives: 1) Representation Stage: It is necessary to jointly capture the spatial and temporal correlations to achieve more accurate feature extraction. 2) Classification Stage: Intuitively, representations of the same activities should be similar. Therefore, using a distance metric which can infer the type of an input segment from labels of the most similar prototype is likely to make the classification module less susceptible to the preciseness of the data representations (compared to neural network based classification). 3) Training Stage: the subject variance can be explicitly modeled and minimized in the training stage to enhance the generalization ability of the approach.

The recent work in HAR has moved towards designing deep learning models for more accurate recognition, given the exceptional representation ability of deep learning techniques. Most deep learning-based HAR methods focus on capturing the temporal correlations in the sensory streams. Jian Bo et al. [16] tackle the problem with convolutional neural networks, in which the convolution and pooling filters are designed along the temporal dimensions to process the readings of all sensors. Their work can capture long-term temporal correlation by stacking multiple CNN layers. Ordóñez et al. [12] further extend this model to DeepConvLSTM by integrating LSTM after CNN layers. The proposed DeepConvLSTM framework contains four CNN layers and two LSTM layers to capture the short-term and long-term temporal correlations, separately. One drawback of the DeepConvLSTM is that it potentially assumes the signals in all time steps are relevant and contribute equally to the target activity, which may not true. Murahari et al. [11] propose to solve the problem by integrating the temporal attention module to DeepConvLSTM. The attention module aligns the output vector at the last time step with other vectors at earlier steps to learn a relative importance score for each previous time step. Different from these methods, Guan et al. [8] propose to achieve more robust data representation ability with the ensemble method. They employ the Epoch-wise Bagging scheme in the training procedure and select multiple LSTMs in different training epochs as basic learners to form a powerful model. However, these methods neglect the spatial correlations among the different sensors, which cannot represent the sensory data precisely. Besides, they directly classify the learned representations to activity type with basic NN-based classifier, which could lead to misguided result due to the learning deviation and subject variances in the representations.

The typical scenario for sensor-based HAR involves multiple devices attached to different parts of the human body. Each device carries multiples sensors, e.g., an inertial measurement unit (IMU) typically contains nine sensors: 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer. In this work, we consider each 3-axis device as three sensors for capturing spatial correlations, e.g., 3-axis accelerometer contains x-accelerometer, y-accelerometer, and z-accelerometer. Thus, an IMU with 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer contains nine sensors. Let M be the total number of sensors embedded in multiple body-worn devices, and \(s_i\) (\(1\le i \le M\)) be the reading from the \(i_{th}\) sensor. Then, at each time point, the sensors, together, generate a vector of readings: \(\varvec{s} = [s_1, s_2, ...\,, s_M]^T\). Thus, a segment with the sliding window size \(\varvec{T}\) can be represented by \(\varvec{Seg} = [\varvec{s_1}, \varvec{s_2}, ...\,, \varvec{s_T}]\).

Let there be N potential activities to be recognized, \(\mathcal {C} = \{c_1, c_2, ...\,, c_N\}\), HAR aims to learn a function, \(\mathcal {F}(\varvec{Seg}, \bullet )\), to infer the correct activity label for the given segment, where \(\bullet \) represents all learnable parameters.

In this section, we elaborate our proposed methods for more accurate HAR, which contains three components: a dual-stream representation module to learn more accurate representations of the input segment, a distance-based classification module to recognize human activities, and a cross-subject training strategy to minimizing the subject-divergence.

4.1 Dual-Stream Representation Module

We first introduce the CNN-based dual-stream representation module (DARM) (shown in Fig. 1), which contains a spatial CNN network and a temporal CNN network. The two CNN networks learn two sub-representations capturing the spatial correlations and temporal correlations within the input segment, respectively, which can be regarded as an image of \(M \times T\) (as denoted in Sect. 3). Then the two sub-representations are merged by summing to get the final joint representation of the input segment. Compared to the previous data representation models, the dual-stream representation module is more accurate by encapsulating both spatial and temporal correlations jointly. Besides, it is more light-weight and easy-to-train compared to LSTM-based approaches [8, 11, 12].

Open image in new window

As shown in Fig. 1, the overall architectures of the temporal CNN and spatial CNN are the same. Both of them contain three consecutive CNN blocks to extract prominent patterns in the segment from different perspectives. The difference between the temporal CNN and spatial CNN mainly lays in the size of CNN kernels. More specifically, the temporal CNN applies the CNN kernels with size \(1\,\times \,k_T^l\) in the \(l_{th}\) T-CNN block, which operate the data along the time axis to capture the temporal correlations between different time points. As a contrast, the spatial CNN applies the CNN kernels with size \(k_S^l\,\times \,k_S^l\) in the \(l_{th}\) S-CNN block to capture the spatial correlations between different sensor series. Besides the kernel size, either of the T-CNN block and the S-CNN block comprises a convolutional layer with a rectified linear units (ReLu) activation function, a max_pooling layer, and a batch-normalization layer. The convolutional layer performs the main function of pattern extraction, which employs several kernels of the same shape to filter the input data X and extract meaningful patterns. We calculate a convolution layer with the ReLu activation function as follows:

$$\begin{aligned} X_j^{l+1} = \sigma (\sum _{i=1}^{i=F^l} W_{i,j} \times X_i^{l} + b_j^l) \end{aligned}$$

(1)

where \(X_i^{l}\) is the \(i_{th}\) channel of the input for the \(l_{th}\) convolutional layer, \(F^l\) is the feature map (channel) numbers, \(W_{i,j}\) is the \(j_{th}\) kernel, \(b_j^l\) is the bias and \(\sigma (\cdot )\) is the ReLu function defined as: \(\sigma (X^{l+1}) = max(0, X^{l+1})\). Then, the max pooling layer is employed as the sampling method to down-sampling the extracted representations while keeping the most protrusive patterns. We further integrate the batch-normalization layer to the CNN block to achieve faster and more stable training. The batch-normalization layer normalizes the layer’s input with batch mean and batch variance to force the input of every layer to have approximately the same distribution [10].

4.2 Distance-Based Classification Module

Based on the representation module, we then propose to learn to recognition human activities by distance based classification module (DCAM) (Fig. 2), which is based on the Prototypical Networks [14]. Different from the general HAR process, which first learns a representation for the input segment and then maps the representation to the corresponding activity with classifiers, DCAM first learns a representation for the input segment and a latent prototype representation (a vector) for each class together. The prototypes are used to represent the embedding of each class. Then, DCAM recognizes the segment representation by comparing its similarity with the prototypes, which follows the same idea with the nearest neighbour methods. For clarity, we denote the data used to learning the prototypes as the support set and the segments to be recognized as queries (see Fig. 2). In the training period, both support set and queries come from the training dataset. In the testing phase, we extract the support set from the training dataset and the queries from the testing dataset to avoid the information leakage.

Open image in new window

To learn the prototypes, we randomly select \(N_s\) samples from each class to form the support set for a batch of queries. Then, these support samples are fed into our dual-stream representation module to get their representations. The prototype of each class is the mean vector of the learned representations in the support set belonging to the corresponding class. Take \(f(\cdot )\) denote the transformation of representation module, \(X_i^j\) as the \(i_{th}\) support sample in the \(j_{th}\) class, then the prototype of class j can be calculated as:

$$\begin{aligned} C_j = \frac{1}{N_s} \sum _{i=1}^{N_s}f(X_i^j) \end{aligned}$$

(2)

Similarly, the query instances are also mapped to the embedding space by our representation module. DCAM can then learn a distribution of a query x over classes based on the softmax of its distances to the learned prototypes \(\{C_1, C_2, ...\,, C_N\}\) in the representation space [14]:

$$\begin{aligned} p_f(y = c_j | x) = \frac{exp(-d(f(x), C_j)}{\sum _{j^\prime =1}^{N} exp(-d(f(x), C_{j^\prime }))} \end{aligned}$$

(3)

where \(d(\cdot , \cdot )\) is a distance function to measure the similarity of two given vectors. There are multiple widely used choices for calculating distance in the literature, such as the Cosine distance, Mahalanobis distance, Euclidean distance and so on. In this work, we employ the squared Euclidean distance as the distance function as it is proved to be more effective than others in [14].

The whole model can be easily trained in an end-to-end manner by minimizing the negative log-probability \(-log(p_f(y = c_j | x))\) according to the true label \(c_j\) of the query segment x via back-propagation strategy. Thus, We define the loss function as follows:

$$\begin{aligned} \mathcal {L}_x = d(f(x), C_j) + log(\sum _{j^\prime =1}^{N} exp(-d(f(x), C_{j^\prime }))) \end{aligned}$$

(4)

4.3 Cross-Subject Training

We further propose the cross-subject training strategy to alleviate the influence of subject variances to the representations. Instead of random sampling support samples and queries from the training set, our cross-subject training strategy intentionally select queries from one subject and support set from other subjects for each batch during the training process. Thus, we can decrease the divergence between different subjects in the representation space through training iteration by minimizing the distance between queries representations and prototypes, which are learned from different subjects separately. Besides, the cross-subject training strategy also harmonizes the training stage and testing stage under the subject-independent setting, where the support set from the training dataset and queries from the testing dataset come from different subject inherently. Algorithm 1 describes the method’s overall training procedure.

Open image in new window

In this work, we propose DHARER – a novel human activity recognition scheme based on similarity comparison and ensembled convolutional neural networks to deal with the representation bias and deviation problem. We first design a dual-stream networks based on CNN to represent the sensory streams more accurately by integrating both spatial and temporal correlations. Then, a distance-based classification model is introduced, which classify the segments by comparing their similarity to the learned prototypes of each class in the representation space. Comparing to the NN-based classification module, the distance-based classification model is less susceptible to the bias in the segment representations. Moreover, we propose the cross-subject training strategy to deal with the deviations caused by subject-variance. Extensive experiments on three datasets demonstrate the superior of our proposed method over several strong SOTAs.