Graph Based Skeleton Motion Representation and Similarity Measurement for Action Recognition

  • Pei Wang
  • Chunfeng YuanEmail author
  • Weiming Hu
  • Bing Li
  • Yanning Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9911)


Most of existing skeleton-based representations for action recognition can not effectively capture the spatio-temporal motion characteristics of joints and are not robust enough to noise from depth sensors and estimation errors of joints. In this paper, we propose a novel low-level representation for the motion of each joint through tracking its trajectory and segmenting it into several semantic parts called motionlets. During this process, the disturbance of noise is reduced by trajectory fitting, sampling and segmentation. Then we construct an undirected complete labeled graph to represent a video by combining these motionlets and their spatio-temporal correlations. Furthermore, a new graph kernel called subgraph-pattern graph kernel (SPGK) is proposed to measure the similarity between graphs. Finally, the SPGK is directly used as the kernel of SVM to classify videos. In order to evaluate our method, we perform a series of experiments on several public datasets and our approach achieves a comparable performance to the state-of-the-art approaches.


3D human action recognition Graph kernel Skeleton motion 



This work is partly supported by the 973 basic research program of China (Grant No. 2014CB349303), the Natural Science Foundation of China (Grant No. 61472421, 61472420, 61303086, 61370185, 61472063), the Natural Science Foundation of Guangdong Province (Grant No. S2013010013432, S2013010015940), and the Strategic Priority Research Program of the CAS (Grant No. XDB02070003).


  1. 1.
    Allen, J.F.: Towards a general theory of action and time. Artif. Intell. 23(2), 123–154 (1984)CrossRefzbMATHGoogle Scholar
  2. 2.
    Amor, B.B., Su, J., Srivastava, A.: Action recognition using rate-invariant analysis of skeletal shape trajectories. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 1–13 (2016)CrossRefGoogle Scholar
  3. 3.
    Anirudh, R., Turaga, P., Su, J., Srivastava, A.: Elastic functional coding of human actions: from vector-fields to latent variables. In: CVPR (2015)Google Scholar
  4. 4.
    Batabyal, T., Chattopadhyay, T., Mukherjee, D.P.: Action recognition using joint coordinates of 3d skeleton data. In: ICIP (2015)Google Scholar
  5. 5.
    Cai, X., Zhou, W., Wu, L., Luo, J., Li, H.: Effective active skeleton representation for low latency human action recognition. IEEE Trans. Multimed, 18(2), 141–154 (2016)CrossRefGoogle Scholar
  6. 6.
    Çeliktutan, O., Wolf, C., Sankur, B., Lombardi, E.: Real-time exact graph matching with application in human action recognition. In: Salah, A.A., Ruiz-del-Solar, J., Meriçli, Ç., Oudeyer, P.-Y. (eds.) HBU 2012. LNCS, vol. 7559, pp. 17–28. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-34014-7_2 CrossRefGoogle Scholar
  7. 7.
    Chaaraoui, A.A., Padilla-López, J.R., Flórez-Revuelta, F.: Fusion of skeletal and silhouette-based features for human action recognition with RGB-D devices. In: ICCVW (2013)Google Scholar
  8. 8.
    Chaudhry, R., Ofli, F., Kurillo, G., Bajcsy, R., Vidal, R.: Bio-inspired dynamic 3d discriminative skeletal features for human action recognition. In: CVPRW (2013)Google Scholar
  9. 9.
    Devanne, M., Wannous, H., Berretti, S., Pala, P.: 3-D human action recognition by shape analysis of motion trajectories on riemannian manifold. IEEE T. Cybern. 45(7), 1023–1029 (2015)Google Scholar
  10. 10.
    Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: CVPR (2015)Google Scholar
  11. 11.
    Ellis, C., Masood, S.Z., Tappen, M.F., Laviola, J.J., Sukthankar, R.: Exploring the trade-off between accuracy and observational latency in action recognition. Int. J. Comput. Vis 101(3), 420–436 (2013)CrossRefGoogle Scholar
  12. 12.
    Evangelidis, G., Singh, G., Horaud, R.: Skeletal quads: human action recognition using joint quadruples. In: ICPR (2014)Google Scholar
  13. 13.
    Gaur, U., Zhu, Y., Song, B., Roy-Chowdhury, A.: A string of feature graphs model for recognition of complex activities in natural videos. In: ICCV (2011)Google Scholar
  14. 14.
    Gowayyed, M., Torki, M., Hussein, M., El-Saban, M.: Histogram of oriented displacements (HOD): describing trajectories of human joints for action recognition. In: IJCAI (2013)Google Scholar
  15. 15.
    Hussein, M., Torki, M., Gowayyed, M., El-Saban, M.: Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. In: IJCAI (2013)Google Scholar
  16. 16.
    Johansson, G.: Visual motion perception. Sci. Am. 232(6), 76–88 (1975)CrossRefGoogle Scholar
  17. 17.
    Kläser, A., Marszalek, M., Schmid, C.: A spatio-temporal descriptor based on 3D-gradients. In: BMVC (2008)Google Scholar
  18. 18.
    Li, W., Zhang, Z., Liu, Z.: Action recognition based on a bag of 3D points. In: CVPRW (2010)Google Scholar
  19. 19.
    Mahé, P., Vert, J.P.: Graph kernels based on tree patterns for molecules. Mach. Learn. 75(1), 3–35 (2009)CrossRefGoogle Scholar
  20. 20.
    Perronnin, F., Dance, C.: Fisher kernels on visual vocabularies for image categorization. In: CVPR (2007)Google Scholar
  21. 21.
    Presti, L.L., Cascia, M.L.: 3D skeleton-based human action classification: a survey. Pattern Recogn. 53, 130–147 (2015)CrossRefGoogle Scholar
  22. 22.
    Scovanner, P., Ali, S., Shah, M.: A 3-dimensional sift descriptor and its application to action recognition. In: ACM MM (2007)Google Scholar
  23. 23.
    Seidenari, L., Varano, V., Berretti, S., Bimbo, A.D., Pala, P.: Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses. In: CVPRW (2013)Google Scholar
  24. 24.
    Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from a single depth image. In: CVPR (2011)Google Scholar
  25. 25.
    Slama, R., Wannous, H., Daoudi, M., Srivastava, A.: Accurate 3D action recognition using learning on the grassmann manifold. Pattern Recogn. 48(2), 556–567 (2015)CrossRefGoogle Scholar
  26. 26.
    Tao, L., Vidal, R.: Moving poselets: a discriminative and interpretable skeletal motion representation for action recognition. In: ICCVW (2015)Google Scholar
  27. 27.
    Unser, M., Aldroubi, A., Eden, M.: B-spline signal processing: part II-efficiency design and applications. IEEE Trans. Sig. Process. 41(2), 834–848 (1993)CrossRefzbMATHGoogle Scholar
  28. 28.
    Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3D skeletons as points in a lie group. In: CVPR (2014)Google Scholar
  29. 29.
    Wallraven, C., Caputo, B., Graf, A.: Recognition with local features: the kernel recipe. In: ICCV (2003)Google Scholar
  30. 30.
    Wang, J., Liu, Z., Wu, Y., Yuan, J.: Learning actionlet ensemble for 3D human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 36(5), 914–927 (2014)CrossRefGoogle Scholar
  31. 31.
    Wang, L., Sahbi, H.: Directed acyclic graph kernels for action recognition. In: ICCV (2013)Google Scholar
  32. 32.
    Willems, G., Tuytelaars, T., Gool, L.: An efficient dense and scale-invariant spatio-temporal interest point detector. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 650–663. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88688-4_48 CrossRefGoogle Scholar
  33. 33.
    Xia, L., Chen, C.C., Aggarwal, J.K.: View invariant human action recognition using histograms of 3D joints. In: CVPRW (2012)Google Scholar
  34. 34.
    Ye, M., Zhang, Q., Liang, W., Zhu, J., Yang, R., Gall, J.: A survey on human motion analysis from depth data. Time-of-Flight Depth Imaging 8200, 149–187 (2013)Google Scholar
  35. 35.
    Zanfir, M., Leordeanu, M., Sminchisescu, C.: The moving pose: an efficient 3D kinematics descriptor for low-latency action recognition and detection. In: ICCV (2013)Google Scholar
  36. 36.
    Zhao, R., Martinez, A.: Labeled graph kernel for behavior analysis. IEEE Trans. Pattern Anal. Mach. Intell. 13(9), 1–13 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Pei Wang
    • 1
  • Chunfeng Yuan
    • 1
    Email author
  • Weiming Hu
    • 1
  • Bing Li
    • 1
  • Yanning Zhang
    • 2
  1. 1.CAS Center for Excellence in Brain Science and Intelligence TechnologyNational Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of SciencesBeijingChina
  2. 2.School of Computer ScienceNorthwestern Polytechnical UniversityXi’anChina

Personalised recommendations