Action recognition in depth videos using hierarchical gaussian descriptor

Abstract

In this paper, we propose a new approach based on distribution descriptors for action recognition in depth videos. Our local features are computed from binary patterns which incorporate the shape and motion cues for effective action recognition. Given pixel-level features, our approach estimates video local statistics in a hierarchical manner, where the distribution of pixel-level features and that of frame-level descriptors are modeled using single Gaussians. In this way, our approach constructs video descriptors directly from low-level features without resorting to codebook learning required by Bag-of-features (BoF) based approaches. In order to capture the spatial geometry and temporal order of a video, we use a spatio-temporal pyramid representation for each video. Our approach is validated on six benchmark datasets, i.e. MSRAction3D, MSRGesture3D, DHA, SKIG, UTD-MHAD and CAD-120. The experimental results show that our approach gives good performance on all the datasets. In particular, it achieves state-of-the-art accuracies on DHA, SKIG and UTD-MHAD datasets.

Appendix: Example to Calculate Binary Patterns for Shape And Motion Cues

An example to calculate LVP_shape,α,D(G_r) with α = 0⁰ and D = 1 is given in Fig. 9, where the referenced pixel G_r is marked in red with its depth value. The first out of 8 bits of LVP_shape,α,D(G_r) is calculated from $I^{\prime }_{\alpha ,D}(G_{1,r})$, $I^{\prime }_{\alpha + 45^{0},D}(G_{1,r})$, $I^{\prime }_{\alpha ,D}(G_{r})$ and $I^{\prime }_{\alpha + 45^{0},D}(G_{r})$. Since $I^{\prime }_{\alpha ,D}(G_{1,r})= 7$, $I^{\prime }_{\alpha + 45^{0},D}(G_{1,r})= 1$, $I^{\prime }_{\alpha ,D}(G_{r})=-3$, $I^{\prime }_{\alpha + 45^{0},D}(G_{r})= 1$ and $1 - \frac {1}{-3} \times 7 > 0$, the first bit of $LVP_{shape,\alpha ,D}(G_{r})$ is assigned to 1. Using similar calculations, the obtained binary code LVP_shape,α,D(G_r) is 11010100, which is 212 in decimal form.

Fig. 9

(Best viewed in color) Example to calculate $\protect LVP_{shape,\alpha ,D}(G_{r})$ with α = 0⁰ and D = 1. The numbers in each image patch represent the depth values of pixels. The referenced pixel G_r is marked in red, the neighboring pixels which are involved in the calculation are marked in blue. The two directions for calculating derivatives at related pixels are shown by the arrows

Another example to calculate $LVP_{motion,D_{1},D_{2}}(G_{r})$ with D₁ = − 1 and D₂ = 1 is given in Fig. 10. The first out of 8 bits of $LVP_{motion,D_{1},D_{2}}(G_{r})$ is calculated from $I^{\prime }_{D_{1}}(G_{1,r})= 2$, $I^{\prime }_{D_{2}}(G_{1,r})= 1$, $I^{\prime }_{D_{1}}(G_{r})= 3$ and $I^{\prime }_{D_{2}}(G_{r})=-1$. Since $1 - \frac {-1}{3} \times 2 > 0$, the first bit of $LVP_{motion,D_{1},D_{2}}(G_{r})$ is assigned to 1. The obtained binary code $LVP_{motion,D_{1},D_{2}}(G_{r})$ is 10101101, which is 173 in decimal form.

Fig. 10

(Best viewed in color) Example to calculate $\protect LVP_{motion,D_{1},D_{2}}(G_{r})$ with D₁ = − 1 and D₂ = 1. The numbers in each image patch represent the depth values of pixels. The motion transformations of the referenced pixel G_r are marked in red, those of the neighboring pixels which are involved in the calculation are marked in blue