Abstract
Human action recognition from skeleton data has drawn a lot of attention from researchers due to the availability of thousands of real videos with many challenges. Existing works attempted to model the spatial characteristics and temporal dependencies of 3D joints using dynamic time warping, hand-crafted, and spatial co-occurrence features. However, the representation derived from the spatial stream overemphasizes the temporal information; thus, it yields limited expressive power. Some studies use skeleton sequences as frames to enhance the expressive power of representations but lose the generalization capability because the derived temporal smoothness is specific to a particular dataset. The proposed work uses joint distance maps as a base representation that encodes the spatial and temporal information to color texture images. We increase the expressive power by extracting the feature maps from pre-trained networks on ImageNet to diversify the texture representation and propose a network architecture to model the temporal dependency explicitly. We also explore various fusion strategies to generate diverse representations from the feature maps of the pre-trained networks. The experimental results show that the proposed method achieves the best recognition accuracy when using decision-level fusion with meta-learners (Random Forest). The analysis also reveals that the use of feature-level fusion yields relatively good results in terms of the trade-off, i.e., on par recognition performance with some decision-level fusion strategies while having less tunable parameters. Extensive experimental results and comparative analysis on three benchmark datasets prove that the proposed representation and network not only yield better recognition accuracy but also exhibit stronger generalization capability on multiple datasets.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Amor BB, Su J, Srivastava A (2016) Action recognition using rate-invariant analysis of skeletal shape trajectories. IEEE Trans Pattern Anal Mach Intell 38(1):1–13. https://doi.org/10.1109/TPAMI.2015.2439257
Avola D, Cascio M, Cinque L, Foresti GL, Massaroni C, Rodola E (2020) 2D skeleton-based action recognition via two-branch stacked LSTM-RNNs. IEEE Trans Multimed. https://doi.org/10.1109/TMM.2019.2960588
Baradel F, Wolf C, Mille J (2018) Human activity recognition with pose-driven attention to RGB. In: Proceedings of the 29th British Machine Vision Conference (BMVC), 1–14.
Bengio Y, Delalleau O (2011) On the expressive power of deep architectures. Int Conf Algorithm Learn Theory. https://doi.org/10.1007/978-3-642-24412-4_3
Bilen H, Fernando B, Gavves E, Vedaldi A, Gould S (2016) Dynamic image networks for action recognition. IEEE Conf Comput Vis Pattern Recog (CVPR) 2016:3034–3042. https://doi.org/10.1109/CVPR.2016.331
Bilen H, Fernando B, Gavves E, Vedaldi A (2018) Action recognition with dynamic image networks. IEEE Trans Pattern Anal Mach Intell 40(12):2799–2813. https://doi.org/10.1109/TPAMI.2017.2769085
CarboneraLuvizon D, Tabia H, Picard D (2017) Learning features combination for human action recognition from skeleton sequences. Pattern Recogn Lett 99:13–20. https://doi.org/10.1016/j.patrec.2017.02.001
Cavazza J, Morerio P, Murino V (2017) Scalable and compact 3D action recognition with approximated rbf kernel machines.
Chaudhry R, Ofli F, Kurillo G, Bajcsy R, Vidal R (2013) Bio-inspired dynamic 3D discriminative skeletal features for human action recognition. IEEE Conf Comput Vis Pattern Recogn Worksh. https://doi.org/10.1109/CVPRW.2013.153
Chen C, Zhuang Y, Nie F, Yang Yi, Fei Wu, Xiao J (2011) Learning a 3D human pose distance metric from geometric pose descriptor. IEEE Trans Visual Comput Graphics 17(11):1676–1689. https://doi.org/10.1109/TVCG.2010.272
Cohen N, Sharir O, Shashua A (2016) On the expressive power of deep learning: a tensor analysis. In: 29th Annual Conference on Learning Theory (PMLR), 698–728.
Donahue J, Hendricks LA, Rohrbach M, Venugopalan S, Guadarrama S, Saenko K, Darrell T (2017) Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans Pattern Anal Mach Intell 39(4):677–691. https://doi.org/10.1109/TPAMI.2016.2599174
Feichtenhofer C, Pinz A, Zisserman A (2016a) Convolutional two-stream network fusion for video action recognition. IEEE Conf Comput Vis Pattern Recog (CVPR). https://doi.org/10.1109/CVPR.2016.213
Feichtenhofer C, Pinz A, Zisserman A (2016b) Convolutional two-stream network fusion for video action recognition. IEEE Conf Comput Vis Pattern Recogn (CVPR) 2016:1933–1941. https://doi.org/10.1109/CVPR.2016.213
Fernando B, Gavves E, Oramas MJO, Ghodrati A, Tuytelaars T (2017) Rank pooling for action recognition. IEEE Trans Pattern Anal Mach Intell 39(4):773–787. https://doi.org/10.1109/TPAMI.2016.2558148
Gao X, Hu W, Tang J, Pan P, Liu J, Guo Z (2018) Generalized graph convolutional networks for skeleton-based action recognition.
Gao X, Hu W, Tang J, Liu J, Guo Z (2019) Optimized skeleton-based action recognition via sparsified graph regression. In: Proceedings of the 27th ACM International Conference on Multimedia, 601–610. https://doi.org/10.1145/3343031.3351170
Goli A, Tirkolaee EB, Aydin NS (2021) Fuzzy integrated cell formation and production scheduling considering automated guided vehicles and human factors. IEEE Trans Fuzzy Syst 29(12):3686–3695. https://doi.org/10.1109/TFUZZ.2021.3053838
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2016.90
Hoang V-N, Le T-L, Tran T-H, Hai-Vu, Nguyen V-T (2019) 3D skeleton-based action recognition with convolutional neural networks. Int Conf Multimed Anal Pattern Recogn (MAPR). https://doi.org/10.1109/MAPR.2019.8743545
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. IEEE/CVF Conf Comput Vis Pattern Recogn 2018:7132–7141. https://doi.org/10.1109/CVPR.2018.00745
Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017a) Densely connected convolutional networks. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2017.243
Huang Z, Wan C, Probst T, Gool LV (2017b) Deep learning on lie groups for skeleton-based action recognition. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2017.137
Hussein MA, Torki M, Gowayyed MA, El-Saban M (2013) Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. In: IJCAI, 2466–2472.
Huynh-The T, Hua C-H, Anh TN, Hur T, Bang J, Kim D, Amin MB, Kang BH, Seung H, Shin S-Y, Kim E-S, Lee S (2018) Hierarchical topic modeling with pose-transition feature for action recognition using 3D skeleton data. Inform Sci 444:20–35. https://doi.org/10.1016/j.ins.2018.02.042
Huynh-The T, Hua C-H, Ngo T-T, Kim D-S (2020) Image representation of pose-transition feature for 3D skeleton-based action recognition. Inf Sci 513:112–126. https://doi.org/10.1016/j.ins.2019.10.047
Jin S-Y, Choi H-J (2013) Essential body-joint and atomic action detection for human activity recognition using longest common subsequence algorithm. Asian Conf Comput Vis. https://doi.org/10.1007/978-3-642-37484-5_13
Khowaja SA, Lee S-L (2019) Hybrid and hierarchical fusion networks: a deep cross-modal learning architecture for action recognition. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04578-y
Khowaja SA, Lee S-L (2020) Semantic image networks for human action recognition. Int J Comput Vision 128(2):393–419. https://doi.org/10.1007/s11263-019-01248-3
Khowaja SA, Yahya BN, Lee S-L (2017) Hierarchical classification method based on selective learning of slacked hierarchy for activity recognition systems. Expert Syst Appl 88:165–177. https://doi.org/10.1016/j.eswa.2017.06.040
Khowaja SA, Khuwaja P, Ismaili IA (2018) A framework for retinal vessel segmentation from fundus images using hybrid feature set and hierarchical classification. SIViP. https://doi.org/10.1007/s11760-018-1366-x
Kim TS, Reiter A (2017) Interpretable 3D human action analysis with temporal convolutional networks. IEEE Conf Comput Vis Pattern Recogn Worksh (CVPRW). https://doi.org/10.1109/CVPRW.2017.207
Kingma DP, Ba J (2014) Adam: A Method for Stochastic Optimization. arXiv:1412.6980
Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances In Neural Information Processing Systems (NIPS), 1–9.
Le Q, Sarlos T, Smola A (2013) Fastfood: approximating kernel expansions in loglinear time. In: Proceedings of the 30th International Conference on Machine Learning, Volume 28, 244–252.
Li C, Wang P, Wang S, Hou Y, Li W (2017a) Skeleton-based action recognition using LSTM and CNN. IEEE Int Conf Multimed Expo Worksh (ICMEW). https://doi.org/10.1109/ICMEW.2017.8026287
Li C, Hou Y, Wang P, Li W (2017b) Joint distance maps based action recognition with convolutional neural networks. IEEE Signal Process Lett 24(5):624–628. https://doi.org/10.1109/LSP.2017.2678539
Li Y, Xia R, Liu X (2020) Learning shape and motion representations for view invariant skeleton-based action recognition. Pattern Recogn 103:107293. https://doi.org/10.1016/j.patcog.2020.107293
Li C, Cui Z, Zheng W, Xu C, Yang J (2018) Spatio-temporal graph convolution for skeleton based action recognition. In: Thirty-Second Association for the Advancement of Artificial Intelligence (AAAI), 3482–3489.
Liu W, Liu H, Tao D, Wang Y, Lu K (2015) Multiview Hessian regularized logistic regression for action recognition. Signal Process 110:101–107. https://doi.org/10.1016/j.sigpro.2014.08.002
Liu M, Liu H, Chen C (2017) Enhanced skeleton visualization for view invariant human action recognition. Pattern Recogn 68:346–362. https://doi.org/10.1016/j.patcog.2017.02.030
Liu J, Shahroudy A, Xu D, Kot AC, Wang G (2018a) Skeleton-based action recognition using spatio-temporal LSTM network with trust gates. IEEE Trans Pattern Anal Mach Intell 40(12):3007–3021. https://doi.org/10.1109/TPAMI.2017.2771306
Liu J, Wang G, Duan L-Y, Abdiyeva K, Kot AC (2018b) Skeleton-based human action recognition with global context-aware attention LSTM networks. IEEE Trans Image Process 27(4):1586–1599. https://doi.org/10.1109/TIP.2017.2785279
Liu X, Li Y, Xia R (2020) Adaptive multi-view graph convolutional networks for skeleton-based action recognition. Neurocomputing. https://doi.org/10.1016/j.neucom.2020.03.126
Lohit S, Wang Q, Turaga P (2019) Temporal transformer networks: joint learning of invariant and discriminative time warping. IEEE/CVF Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2019.01271
Lu Z, Pu H, Wang F, Hu Z, Wang L (2017) The expressive power of neural networks: a view from the width. Adv Neural Inform Process Syst (NeurIPS): 1–9.
Mahasseni B, Todorovic S (2016) Regularizing long short term memory with 3d human-skeleton sequences for action recognition. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2016.333
Müller M, Röder T, Clausen M (2005) Efficient content-based retrieval of motion capture data. ACM Trans Graph 24(3):677. https://doi.org/10.1145/1073204.1073247
Naveenkumar M, Domnic S (2020) Deep ensemble network using distance maps and body part features for skeleton based action recognition. Pattern Recogn 100:107125. https://doi.org/10.1016/j.patcog.2019.107125
Novak R, Bahri Y, Abolafia DA, Pennington J, Sohl-Dickstein J (2018) Sensitivity and generalization in neural networks: an empirical study. In: International Conference on Learning Representations, 1–20.
Núñez JC, Cabido R, Pantrigo JJ, Montemayor AS, Vélez JF (2018) Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recogn 76:80–94. https://doi.org/10.1016/j.patcog.2017.10.033
Ofli F, Chaudhry R, Kurillo G, Vidal R, Bajcsy R (2014) Sequence of the most informative joints (SMIJ): a new representation for human skeletal action recognition. J Vis Commun Image Represent 25(1):24–38. https://doi.org/10.1016/j.jvcir.2013.04.007
Ohn-Bar E, Trivedi MM (2013) Joint angles similarities and HOG2 for action recognition. IEEE Conf Comput Vis Pattern Recogn Worksh. https://doi.org/10.1109/CVPRW.2013.76
Pham H-H, Khoudour L, Crouzil A, Zegers P, Velastin SA (2018) Exploiting deep residual networks for human action recognition from skeletal data. Comput vis Image Underst 170:51–66. https://doi.org/10.1016/j.cviu.2018.03.003
Pham HH, Salmane H, Khoudour L, Crouzil A, Zegers P, Velastin SA (2019) Spatio-temporal image representation of 3D skeletal movements for view-invariant action recognition with deep convolutional neural networks. Sensors 19(8):1932. https://doi.org/10.3390/s19081932
Raghu M, Poole B, Kleinberg J, Ganguli S, Dickstein JS (2017) On the expressive power of deep neural networks. In: Proceedings of the 34 Th International Conference on Machine Learning (ICML), 1–24.
Schuldt C, Laptev I, Caputo B (2004) Recognizing human actions: a local SVM approach. In: Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, Vol. 3, pp 32–36. https://doi.org/10.1109/ICPR.2004.1334462
Seidenari L, Varano V, Berretti S, Del Bimbo A, Pala P (2013) Recognizing actions from depth cameras as weakly aligned multi-part bag-of-poses. IEEE Conf Comput Vis Pattern Recogn Worksh. https://doi.org/10.1109/CVPRW.2013.77
Serra T, Tjandraatmadja C, Ramalingam S (2018) Bouding and counting linear regions of deep neural networks. In: Proceedings of the 35th International Conference on Machine Learning, 4558–4566.
Shahroudy A, Liu J, Ng T-T, Wang G (2016a) NTU RGB+D: a large scale dataset for 3D human activity analysis. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2016.115
Shahroudy A, Ng T-T, Yang Q, Wang G (2016b) Multimodal multipart learning for action recognition in depth videos. IEEE Trans Pattern Anal Mach Intell 38(10):2123–2129. https://doi.org/10.1109/TPAMI.2015.2505295
Shi Z, Kim T-K (2017) Learning and refining of privileged information-based RNNs for action recognition from depth sequences. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2017.498
Shi Y, Tian Y, Wang Y, Huang T (2017) Sequential deep trajectory descriptor for action recognition with three-stream CNN. IEEE Trans Multimed 19(7):1510–1520. https://doi.org/10.1109/TMM.2017.2666540
Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Adv Neural Inform Process Syst:1–9.
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2015.7298594
Tao L, Vidal R (2015) Moving poselets: a discriminative and interpretable skeletal motion representation for action recognition. IEEE Int Conf Comput Vis Workshop (ICCVW). https://doi.org/10.1109/ICCVW.2015.48
Varol G, Laptev I, Schmid C (2018) Long-term temporal convolutions for action recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1510–1517. https://doi.org/10.1109/TPAMI.2017.2712608
Velastin S, Pham HH, Khoudour L, Crouzil A, Zegers P (2018) Learning to recognize 3D human action from a new skeleton-based representation using deep convolutional neural networks. IET Comput Vision. https://doi.org/10.1049/iet-cvi.2018.5014
Vemulapalli R, Arrate F, Chellappa R (2014) Human action recognition by representing 3D skeletons as points in a lie group. IEEE Conf Comput Vis Pattern Recogn. https://doi.org/10.1109/CVPR.2014.82
Vinagre M, Aranda J, Casals A (2015) A new relational geometric feature for human action recognition. Inform Control Automat Robot. https://doi.org/10.1007/978-3-319-10891-9_15
Wang J, Liu Z, Ying Wu, Yuan J (2012) Mining actionlet ensemble for action recognition with depth cameras. IEEE Conf Comput Vis Pattern Recogn. https://doi.org/10.1109/CVPR.2012.6247813
Wang J, Liu Z, Wu Y, Yuan J (2014) Learning actionlet ensemble for 3D human action recognition. IEEE Trans Pattern Anal Mach Intell 36(5):914–927. https://doi.org/10.1109/TPAMI.2013.198
Wang P, Li W, Gao Z, Zhang J, Tang C, Ogunbona PO (2016a) Action recognition from depth maps using deep convolutional neural networks. IEEE Trans Hum Mach Syst 46(4):498–509. https://doi.org/10.1109/THMS.2015.2504550
Wang P, Li Z, Hou Y, Li W (2016b) Action recognition based on joint trajectory maps using convolutional neural networks. In: Proceedings of the ACM on Multimedia Conference - MM ’16, 102–106. https://doi.org/10.1145/2964284.2967191
Wang P, Li W, Li C, Hou Y (2018) Action recognition based on joint trajectory maps with convolutional neural networks. Knowl Based Syst 158:43–53. https://doi.org/10.1016/j.knosys.2018.05.029
Wang C, Kong C, Lucey S (2019) Distill knowledge from NRSfM for weakly supervised 3D pose learning. IEEE/CVF Int Conf Comput Vis (ICCV). https://doi.org/10.1109/ICCV.2019.00083
Wu Z, Jiang Y-G, Wang X, Ye H, Xue X (2016) Multi-stream multi-class fusion of deep networks for video classification. In: ACM on Multimedia Conference - MM ’16, 791–800. https://doi.org/10.1145/2964284.2964328
Wu B, Xu C, Dai X, Wan A, Zhang P, Yan Z, Tomizuka M, Gonzalez J, Keutzer K, Vajda P (2020) Visual transformers: token-based image representation and processing for computer vision.
Xia L, Chen C-C, Aggarwal JK (2012) View invariant human action recognition using histograms of 3D joints. IEEE Comput Soc Conf Comput Vis Pattern Recogn Workshops. https://doi.org/10.1109/CVPRW.2012.6239233
Xu J, Tasaka K, Yanagihara H (2018a) Beyond two-stream: skeleton-based three-stream networks for action recognition in videos. In: 24th International Conference on Pattern Recognition (ICPR), 1567–1573. https://doi.org/10.1109/ICPR.2018.8546165
Xu Y, Cheng J, Wang L, Xia H, Liu F, Tao D (2018b) Ensemble one-dimensional convolution neural networks for skeleton-based action recognition. IEEE Signal Process Lett 25(7):1044–1048. https://doi.org/10.1109/LSP.2018.2841649
Yang Y, Deng C, Tao D, Zhang S, Liu W, Gao X (2016) Latent max-margin multitask learning with skelets for 3-D action recognition. IEEE Trans Cybern 47(2):1–10. https://doi.org/10.1109/TCYB.2016.2519448
Yao A, Gall J, Fanelli G, Gool LV (2011) Does human action recognition benefit from pose estimation? In: Procedings of the British Machine Vision Conference 2011, 67.1–67.11. https://doi.org/10.5244/C.25.67
Yong D, Wang W, Wang L (2015) Hierarchical recurrent neural network for skeleton based action recognition. IEEE Conf Comput Vis Pattern Recogn (CVPR). https://doi.org/10.1109/CVPR.2015.7298714
Yun K, Honorio J, Chattopadhyay D, Berg TL, Samaras D (2012) Two-person interaction detection using body-pose features and multiple instance learning. IEEE Comput Soc Conf Comput Vis Pattern Recogn Workshops. https://doi.org/10.1109/CVPRW.2012.6239234
Zach C, Pock T, Bischof H (2007) A duality based approach for realtime TV-L 1 optical flow. Pattern Recogn. https://doi.org/10.1007/978-3-540-74936-3_22
Zhang X, Wu D (2020) Empirical studies on the properties of linear regions in deep neural networks. In: International Conference on Learning Representations, 1–17.
Zhang S, Yang Y, Xiao J, Liu X, Yang Y, Xie D, Zhuang Y (2018) Fusing geometric features for skeleton-based action recognition using multilayer LSTM networks. IEEE Trans Multimed 20(9):2330–2343. https://doi.org/10.1109/TMM.2018.2802648
Zhu W, Lan C, Xing J, Zeng W, Li Y, Shen L, Xie X (2016) Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. In: Thirteith AAAI Conference on Artificial Intelligence, 3697–3703.
Acknowledgements
This work was supported by Hankuk University of Foreign Studies Research Fund and by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1F1A1060244).
Funding
This work was supported by Hankuk University of Foreign Studies Research Fund and by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1F1A1060244).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Khowaja, S.A., Lee, SL. Skeleton-based human action recognition with sequential convolutional-LSTM networks and fusion strategies. J Ambient Intell Human Comput 13, 3729–3746 (2022). https://doi.org/10.1007/s12652-022-03848-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-022-03848-3