Skip to main content
SpringerLink
Log in

Two-stream adaptive-attentional subgraph convolution networks for skeleton-based action recognition

  • 1193: Intelligent Processing of Multimedia Signals
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recently, skeleton-based action recognition has modeled the human skeleton as a graph convolution network (GCN), and has achieved remarkable results. However, most of the methods convolute directly on the whole graph, neglecting that the human skeleton is made up of multiple body parts, which cannot accomplish the task well. We recognize that the physical property of bones (i.e., length and direction) can provide identifiable information which helps effectively to build the multi-level network structure. As the existing methods treat the channel domain and the spatial domain with equal importance, many computing resources are wasted on neglectable features. In our paper, we modify the Convolution Block Attention Module (CBAM) and apply it to the adaptive network. By capturing the implicit weighted information in the channel domain and spatial domain, the network can focus more attention on the key channels and nodes. A new two-stream adaptive-attentional subgraph convolution network (2s-AASGCN) is proposed to extract features in the spatio-temporal domain. We validate 2s-AASGCN on two skeleton datasets, i.e., NTU-RGB+D60 and NTU-RGB+D120. Our model achieves excellent results on these two datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Bo L, Dai Y, Cheng X, Chen H, Yi L, He M (2017) Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN. In: 2017 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp 601–604

  2. Du Y, Wang W, Wang L (2015) Hierarchical recurrent neural network for skeleton based action recognition. In: Computer Vision and Pattern Recognition IEEE

  3. Fernando B, Gavves E, Oramas JM, Ghodrati A, Tuytelaars T (2015) Modeling video evolution for action recognition. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) IEEE Computer Society, pp 5378–5387

  4. Gao X, Hu W, Tang J, Liu J, Guo Z-M (2019) Optimized Skeleton-based action recognition via sparsified graph regression. the 27th ACM International Conference ACM

  5. Gaur U, Zhu Y, Song B, Roy-Chowdhury A (2011) A string of feature graphs model for recognition of complex activities in natural videos. In: 2011 International Conference on Computer Vision, pp 2595–2602

  6. Jie H, Li S, Albanie S, Sun G, Vedaldi A (2018) Gather-excite: exploiting feature context in convolutional neural networks. In: Advances in neural information processing systems, pp 9401–9411

  7. Ke Q, Bennamoun M, An S (2017) A new representation of skeleton sequences for 3d action recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3288–3297

  8. Ke Q, Bennamoun M, An S, Sohel F, Boussaid F (2018) Learning clip representations for skeleton-based 3d action recognition. IEEE Trans Image Process, pp 2842–2855

  9. Ke C, Zhang Y, He X, Chen W, Cheng J, Hangqing L (2020) Skeleton-based action recognition with shift graph convolutional network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 183–192

  10. Kim TS, Reiter A (2017) Interpretable 3d human action analysis with temporal convolutional networks. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 1623–1631

  11. Li M, Chen S, Chen X, Ya Z, Wang Y, Qi T (2019) Actional-structural graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3595–3603

  12. Li S, Li W, Cook C, Ce Z, Gao Y (2018) Independently recurrent neural network (indrnn): building a longer and deeper rnn. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5457–5466

  13. Liao S, Lyons TJ, Yang W, Ni H (2019) Learning stochastic differential equations using RNN with log signature features. arXiv preprint arXiv

  14. Liu H, Juanhui T, Liu M (2017) Two-stream 3d convolutional neural network for skeleton-based action recognition. arXiv preprint arXiv:1705.08106

  15. Liu M, Liu H, Chen C (2017) Enhanced skeleton visualization for view invariant human action recognition Pattern Recogn, pp 346–362

  16. Liu J, Shahroudy A, Dong X, Wang G (2016) Spatio-temporal lstm with trust gates for 3d human action recognition. In: European conference on computer vision, pp 816–833

  17. Liu J, Shahroudy A, Perez ML, Wang G, Duan L-Y, Chichung AK (2019) A large-scale benchmark for 3d human activity understanding. IEEE transactions on pattern analysis and machine intelligence

  18. Liu J, Wang G, Duan L-Y (2017) Skeleton-based human action recognition with global context-aware attention LSTM networks. IEEE Trans Image Process, pp 1586–1599

  19. Liu J, Wang G, Hu P, Duan L-Y, CKot A (2017) Global context-aware attention LSTM Networks for 3D action recognition. IEEE Conference on Computer Vision and Pattern Recognition IEEE

  20. Liu M, Yuan J (2018) Recognizing human actions as the evolution of pose estimation maps. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1159–1168

  21. Liu Z, Zhang H, Chen Z, Wang Z, Ouyang W (2020) Disentangling and unifying graph convolutions for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 143–152

  22. Peng W, Hong X, Chen H, Zhao G (2020) Learning graph convolutional network for skeleton-based human action recognition by neural searching. In: AAAI, pp 2669–2676

  23. Shahroudy A, Liu J, Ng T-T, Wang G (2016) Ntu rgb+ d: a large scale dataset for 3d human activity analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1010–1019

  24. Shi H, Meng X, Hwang K-S, Cai B-Y (2020) Behavior fusion for deep reinforcement learning. ISA transactions, pp. 434–444

  25. Shi H, Wu H, Hwang K-S (2020) Adaptive image-based visual servoing using reinforcement learning with fuzzy state coding. IEEE Transactions on Fuzzy Systems

  26. Shi L, Zhang Y, Cheng J, Hanqing L (2019) Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 12026–12035

  27. Shi L, Zhang Y, Cheng J, Hanqing LS (2019) Skeleton-based action recognition with directed graph neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7912–7921

  28. Shi L, Zhang Y, Cheng J, Hanqing L (2019) Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Trans Image Process, pp 9532–9545

  29. Si C, Chen W, Wang W, Wang L, Tan T (2019) An attention enhanced graph convolutional lstm network for skeleton-based action recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1227–1236

  30. Song S, Lan C, Xing J, Zeng W, Liu J (2015) An end-to-end spatio-temporal attention model for human action recognition from skeleton data. arXiv preprint arXiv:1611.06067

  31. Sudha MR, Sriraghav K, Abisheck SS, Jacob SG, Manisha S (2017) Approaches and applications of virtual reality and gesture recognition: a review. International Journal of Ambient Computing and Intelligence (IJACI), pp. 1–18

  32. Tang Y, Yi T, Jiwen L, Li P, Zhou J (2018) Deep progressive reinforcement learning for skeleton-based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5323–5332

  33. Thakkar K, Narayanan PJ (2018) Part-based graph convolutional network for action recognition. arXiv preprint arXiv:180.04983

  34. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  35. Vemulapalli R, Arrate F, Chellappa R (2014) Human action recognition by representing 3d skeletons as points in a lie group. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 588–595

  36. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3156–3164

  37. Woo S, Park J, Lee J-Y, In SK (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  38. Yan S, Xiong Y, Lin D (2018) Spatial temporal graph convolutional networks for skeleton-based action recognition. arXiv preprint arXiv:1801.07455

  39. Zhang Xikun, Chang X u, Tao Dacheng (2020) Context aware graph convolution for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 14333–14342

  40. Zhang P, Lan C, Xing J, Zeng W (2018) View adaptive neural networks for high performance skeleton-based human action recognition. In: IEEE Transactions on Pattern Analysis and Machine Intelligence

  41. Zhang P, Lan C, Xing J, Zeng W, Xue J, Zheng N (2017) View adaptive recurrent neural networks for high performance human action recognition from skeleton data

  42. Zhang P, Lan C, Zeng W, Xing J, Xue J, Zheng N (2020) Semantics-guided neural networks for efficient skeleton-based human action recognition. In: Proc IEEE Conf. Comput. Vis. Pattern Recognit, pp 1112–1121

Download references

Acknowledgment

This work was supported by the Natural Science Foundation of Hebei Province Grant No. F2018203390, Qinhuangdao City Science and Technology Research and Development Plan Grant No.202003B043 and Xinjiang Uygur Autonomous Region University Scientific Research Project (Key Natural Science Project) XJEDU2021I029. The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the paper.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yanshan University, Qinhuangdao, 066004, China

    Xianshan Li, Fengchan Meng, Fengda Zhao, Dingding Guo, Fengwei Lou & Rong Jing

  2. Xinjiang University of Science and Technology, Korla, 841000, China

    Fengda Zhao

  3. The Key Laboratory for Software Engineering of Hebei Province, Yanshan University, Qinhuangdao, 066004, China

    Xianshan Li, Fengchan Meng, Fengda Zhao, Dingding Guo, Fengwei Lou & Rong Jing

Authors
  1. Xianshan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fengchan Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fengda Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dingding Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fengwei Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rong Jing
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fengda Zhao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Meng, F., Zhao, F. et al. Two-stream adaptive-attentional subgraph convolution networks for skeleton-based action recognition. Multimed Tools Appl 81, 4821–4838 (2022). https://doi.org/10.1007/s11042-021-11026-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11026-4

Keywords

Search

Navigation