Gait representation and recognition from temporal co-occurrence of flow fields

Abstract

This paper proposes a new gait representation that encodes the dynamics of a gait period through a 2D array of 17-bin histograms. Every histogram models the co-occurrence of optical flow states at every pixel of the normalized template that bounds the silhouette of a target subject. Five flow states (up, down, left, right, null) are considered. The first histogram bin counts the number of frames over the gait period in which the optical flow for the corresponding pixel is null. In turn, each of the remaining 16 bins represents a pair of flow states and counts the number of frames in which the optical flow vector has changed from one state to the other during the gait period. Experimental results show that this representation is significantly more discriminant than previous proposals that only consider the magnitude and instantaneous direction of optical flow, especially as the walking direction gets closer to the viewing direction, which is where state-of-the-art gait recognition methods yield the lowest performance. The dimensionality of that gait representation is reduced through principal component analysis. Finally, gait recognition is performed through supervised classification by means of support vector machines. Experimental results using the public CMU MoBo and AVAMVG datasets show that the proposed approach is advantageous over state-of-the-art gait representation methods.

Appendix

Each of the 17-bin histograms models a co-occurrence of optical flow states between a pair of templates separated by m frames, $T_i$ and $T_j$, where $j = (i+m)\%n$, $0< m < n$ and $0\le i,j < n$, with n being the number of frames in a gait period. Those bins are noted as: $HV _i$, $HR _i$, $HL _i$, $LR _i$, $LL _i$, $LH _i$, $RR _i$, $RL _i$, $RH _i$, $VU _i$, $VD _i$, $UU _i$, $UD _i$, $UV _i$, $DU _i$, $DD _i$, $DV _i$.

$$\begin{aligned} HV _i(x,y)= & {} H_i(x,y)\times V_j(x,y), \end{aligned}$$

(13)

$$\begin{aligned} HR _i(x,y)= & {} H_i(x,y)\times R_j(x,y), \end{aligned}$$

(14)

$$\begin{aligned} HL _i(x,y)= & {} H_i(x,y)\times L_j(x,y), \end{aligned}$$

(15)

$$\begin{aligned} LR _i(x,y)= & {} L_i(x,y)\times R_j(x,y), \end{aligned}$$

(16)

$$\begin{aligned} LL _i(x,y)= & {} L_i(x,y)\times L_j(x,y), \end{aligned}$$

(17)

$$\begin{aligned} LH _i(x,y)= & {} L_i(x,y)\times H_j(x,y), \end{aligned}$$

(18)

$$\begin{aligned} RR _i(x,y)= & {} R_i(x,y)\times R_j(x,y), \end{aligned}$$

(19)

$$\begin{aligned} RL _i(x,y)= & {} R_i(x,y)\times L_j(x,y), \end{aligned}$$

(20)

$$\begin{aligned} RH _i(x,y)= & {} R_i(x,y)\times H_j(x,y), \end{aligned}$$

(21)

$$\begin{aligned} VU _i(x,y)= & {} V_i(x,y)\times U_j(x,y), \end{aligned}$$

(22)

$$\begin{aligned} VD _i(x,y)= & {} V_i(x,y)\times D_j(x,y), \end{aligned}$$

(23)

$$\begin{aligned} UU _i(x,y)= & {} U_i(x,y)\times U_j(x,y), \end{aligned}$$

(24)

$$\begin{aligned} UD _i(x,y)= & {} U_i(x,y)\times D_i(x,y), \end{aligned}$$

(25)

$$\begin{aligned} UV _i(x,y)= & {} U_i(x,y)\times V_j(x,y), \end{aligned}$$

(26)

$$\begin{aligned} DU _i(x,y)= & {} D_i(x,y)\times U_j(x,y), \end{aligned}$$

(27)

$$\begin{aligned} DD _i(x,y)= & {} D_i(x,y)\times D_j(x,y), \end{aligned}$$

(28)

$$\begin{aligned} DV _i(x,y)= & {} D_i(x,y)\times V_j(x,y). \end{aligned}$$

(29)

Finally, an accumulation $h \times w$ histogram of 17 bins is computed. The 17 bins can be defined as:

$$\begin{aligned} HV (x,y)= & {} \sum _{i=0}^{n-1} HV _i(x,y), \end{aligned}$$

(30)

$$\begin{aligned} HR (x,y)= & {} \sum _{i=0}^{n-1} HR _i(x,y), \end{aligned}$$

(31)

$$\begin{aligned} HL (x,y)= & {} \sum _{i=0}^{n-1} HL _i(x,y), \end{aligned}$$

(32)

$$\begin{aligned} LR (x,y)= & {} \sum _{i=0}^{n-1} LR _i(x,y), \end{aligned}$$

(33)

$$\begin{aligned} LL (x,y)= & {} \sum _{i=0}^{n-1} LL _i(x,y), \end{aligned}$$

(34)

$$\begin{aligned} LH (x,y)= & {} \sum _{i=0}^{n-1} LH _i(x,y), \end{aligned}$$

(35)

$$\begin{aligned} RR (x,y)= & {} \sum _{i=0}^{n-1} RR _i(x,y), \end{aligned}$$

(36)

$$\begin{aligned} RL (x,y)= & {} \sum _{i=0}^{n-1} RL _i(x,y), \end{aligned}$$

(37)

$$\begin{aligned} RH (x,y)= & {} \sum _{i=0}^{n-1} RH _i(x,y), \end{aligned}$$

(38)

$$\begin{aligned} VU (x,y)= & {} \sum _{i=0}^{n-1} VU _i(x,y), \end{aligned}$$

(39)

$$\begin{aligned} VD (x,y)= & {} \sum _{i=0}^{n-1} VD _i(x,y), \end{aligned}$$

(40)

$$\begin{aligned} UU (x,y)= & {} \sum _{i=0}^{n-1} UU _i(x,y), \end{aligned}$$

(41)

$$\begin{aligned} UD (x,y)= & {} \sum _{i=0}^{n-1} UD _i(x,y), \end{aligned}$$

(42)

$$\begin{aligned} UV (x,y)= & {} \sum _{i=0}^{n-1} UV _i(x,y), \end{aligned}$$

(43)

$$\begin{aligned} DU (x,y)= & {} \sum _{i=0}^{n-1} DU _i(x,y), \end{aligned}$$

(44)

$$\begin{aligned} DD (x,y)= & {} \sum _{i=0}^{n-1} DD _i(x,y), \end{aligned}$$

(45)

$$\begin{aligned} DV (x,y)= & {} \sum _{i=0}^{n-1} DV _i(x,y). \end{aligned}$$

(46)