View Transfer on Human Skeleton Pose: Automatically Disentangle the View-Variant and View-Invariant Information for Pose Representation Learning

Abstract

Learning a good pose representation is significant for many applications, such as human pose estimation and action recognition. However, the representations learned by most approaches are not intrinsic and their transferability in different datasets and different tasks is limited. In this paper, we introduce a method to learn a versatile representation, which is capable of recovering unseen corrupted skeletons, being applied to the human action recognition, and transferring pose from one view to another view without knowing the relationships of cameras. To this end, a sequential bidirectional recursive network (SeBiReNet) is proposed for modeling kinematic dependency between skeleton joints. Utilizing the SeBiReNet as the core module, a denoising autoencoder is designed to learn intrinsic pose features through the task of recovering corrupted skeletons. Instead of only extracting the view-invariant feature as many other methods, we disentangle the view-invariant feature from the view-variant feature in the latent space and use them together as a representation of the human pose. For a better feature disentanglement, an adversarial augmentation strategy is proposed and applied to the denoising autoencoder. Disentanglement of view-variant and view-invariant features enables us to realize view transfer on 3D poses. Extensive experiments on different datasets and different tasks verify the effectiveness and versatility of the learned representation.

Appendices

Appendix A More Details About the SeBiReNet

1.1 A.1 Transformation Between the Human Skeleton and Tree Structure

As shown in Fig 12, the left one is a human skeleton model with 25 joints as defined by the Kinect sensor. The joint 0 (hip center) is generally selected as the root joint and its coordinate is set to (0,0,0). By reordering those joints but keeping the connection relationship between them, we can obtain a tree structure of the human body shown as the middle one (finger joints 7, 10, 21, 22, 23, 24 and feet joints 15, 19 are discarded). By replacing the joints with neural cells, a conventional recursive neural network corresponding to the human body is obtained as shown in the right side.

Fig. 12

Transformation from the human skeleton to the tree structure network

1.2 A.2 Relationship Between the Root Node and the Leaf Node

In the proposed SeBiReNet, each node actually corresponds to a joint in the human body. Therefore, the relationship between the root node and the leaf node is similar to the relationship between the root joint and the end joint. Usually, human motion transmits from the root joint to the end joint. However, to determine the position of a joint, both its parent joint and child joints should be considered. To this end, by imitating the physical forward kinematics and inverse kinematic process, we design our SeBiReNet as shown in the Fig. 2 in the Sect. 3.1.

1.3 A.3 Differences with the Other Tree Structures

The conventional recursive neural network shown in Fig. 12 is one of the most widely used tree structure network, especially in the text or language analysis field. In conventional recursive neural network, information flows from leaf nodes to root node. Hence, it’s strong at summarizing the input information and getting a semantic understanding. Compared to the recursive neural network or other tree structures, our SeBiReNet differs with them in the following aspects:

• a. Structure: The proposed SeBiReNet sequentially connects two tree structures that have opposite information flow directions. As shown in Fig. 2, we call the left part (a conventional recursive NN) recursive subnetwork and the right part diffuse subnetwork according to the information flow direction. The recursive subnetwork imitates the inverse kinematic process and the diffuse subnetwork imitates the forward kinematic process. Compared to other tree structures in general, the SeBiReNet has an iterative and directed inference process with ordered dependency pairs.

• b. Hidden states: We share the hidden states between the corresponding nodes in the two subnetworks considering the fact that they represent a same joint in the human body. In this manner, the result of the recursive process stored in the shared space can be refined by the diffuse subnetwork, and vice versa. Therefore, the whole SeBiReNet is able to run recurrently to achieve a better result.

• c. Outputs: The output of SeBiReNet is generated from the shared hidden states of all joints. This is different from the recursive network that only considers the output of the root node. It’s also different from the regular fusion design that concatenates the results of the two subnetworks.

1.4 A.4 Inference Process of the SeBiReNet

1.5 A.5 Network used for the Ablation Study in Sect. 5.3.1

Appendix B Pose Recovery Results on Unseen Action Datasets

Appendix C Pose View Transfer Based on the Feature Disentanglement in the Latent Space

Table 7 Results of the action clustering

Appendix D Unsupervised Classification with Clustering Methods

To demonstrate the effectiveness of the learned representation more rigorously, we compared the clustering results achieved from the learned pose representation and the joint coordinates using different clustering methods, as shown in Table 7. Different from the supervised method, it’s difficult to evaluate the output of clustering methods in an intuitive manner. The purity and adjusted rand index (ARI) are adopted as the measure of clustering quality. Purity is a measure of the extent to which clusters contains a single class. The adjusted rand index (ARI$\in [-1,1]$) measures how better the clustering results coincide with the true classes. Though both of them can reflect the clustering quality, ARI is better in measuring the correctness of unsupervised classification to some extent.

In Table 7, we compared four different clustering methods: K-means, Mean shift, Hierarchical clustering and Gaussian Mixture Model (GMM). For K-means, Hierarchical clustering and GMM methods, we pre-set the number of clusters (k) equal to the true number of classes. Mean shift achieves the number of clusters automatically and the number of clusters achieved is denoted as $k^*$. As K-means and GMM are sensitive to the initialization of parameters, we report the average results and its standard deviation. GMM is not suitable for processing high dimensional data because a large scale of memory is needed. To make the GMM method computable, we use the principle analysis (PCA) to lower the dimension of action representation from $TimeSteps*f_{vi}$ to 100.

On the Northwestern-UCLA dataset, the purity and ARI achieved from the learned representation are much higher than the results of using joint coordinates in K-means, Hierarchical clustering, and GMM methods. As the attained clusters are much more than the true classes, the purity and ARI are meaningless in the Mean Shift method. However, the cluster number obtained automatically by the Mean Shift method from the learned representation is much less than the 234 clusters based on the joint coordinates. Compared to the joint coordinates, the learned representation shows a much better performance in unsupervised action clustering.

Different from the learning methods based on LSTM, clustering methods cannot handle the time variation of an action. Hence, the clustering result becomes bad when action lengths vary a lot in a large dataset, such as the NTU RGB+D dataset. As Table 7 shows, both the clustering results based on the learned representation and the joint coordinates are not good on the NTU RGB+D dataset. While the ARI achieved from the learned representation is still higher than the one achieved from joint coordinates, which indicates that the learned representation leads to a better action classification. The number of clusters attained using the learned representation is also much less than the number of clusters using the joint coordinates in Mean shift method.