Abstract
Human behavior recognition is a hot research topic in the field of computer vision, and a complete behavior recognition usually includes human detection, human tracking and behavior recognition. At present, the two tasks of human tracking and abnormal behavior recognition based on deep learning are mostly executed separately, and the related feature information in the two tasks cannot be fully utilized, resulting in high time cost and resource consumption of the final abnormal behavior recognition algorithm. The problem greatly limits the widespread application of abnormal behavior recognition. In order to improve the performance of the algorithm a novel model for abnormal behaviors recognition based on human target tracking is proposed, which implements the process of recognizing abnormal behaviors after human target tracking through feature sharing. First, the real-time multi-domain convolutional neural network is improved by introducing a spatial attention mechanism to improve its tracking of a particular human body in a video series. Then the output of the convolutional layer in MDnet is used as the input of the abnormal behavior recognition network, and these features are combined with CNN and LSTM to realize human abnormal behavior recognition. During the network training process, a multi-task learning approach was used to train a model for human tracking and behaviour recognition. Six types of abnormal behaviors selected on the CASIA Behavioural Analytics dataset and 12 types of behaviours selected on the NTU database are used to train and test the network model. According to test results, the proposed model is capable of tracking human targets precisely and in real time (26 frames per second). The proposed model can also distinguish abnormal behaviors of tracking targets with a recognition rate of 92.1%. The human features obtained in the tracking model is used as the input of the abnormal behavior recognition network, so the feature sharing of tracking and recognition is achieved, and a complete abnormal behavior recognition framework including target tracking, feature extraction, and behavior recognition is established. There is great practical significance to the proposed method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Some or all data, models, or code generated or used during the study are available from the corresponding author by request.
References
Bhat, G., Johnander, J., Danelljan, M., Khan, F.S., Felsberg, M.: Unveiling the power of deep tracking. ArXiv abs/1804.06833, 493–509 (2018)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4733 (2017). https://doi.org/10.1109/CVPR.2017.502
Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., Lu, H.: Transformer tracking. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8122–8131 (2021)
Ge, S., Luo, Z., Zhang, C., Hua, Y., Tao, D.: Distilling channels for efficient deep tracking. IEEE Trans. Image Process. 29, 2610–2621 (2020)
Hu, J., Qi, Y., Wang, J.: Student online classroom behavior recognition based on spatio-temporal graph convolutional network. Optoelectron. Laser 33, 149–156 (2022)
Khaled, N., Marey, M.A.E.-R., Aref, M.M.: Temporal action detection with fused two-stream 3d residual neural networks and bi-directional lstm. 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), 130–140 (2019). https://doi.org/10.1109/ICICIS46948.2019.9014707
Li, R., Li, S.: Human behavior recognition based on attention mechanism. In: 2020 International Conference on Artificial Intelligence and Education (ICAIE), pp. 103–107 (2020). https://doi.org/10.1109/ICAIE50891.2020.00031
Liao, Z., Hu, H., Zhang, J., Yin, C.: Residual attention unit for action recognition. Comput. Vis. Image Underst. 189 (2019). https://doi.org/10.1016/j.cviu.2019.102821
Liu, Q., Zhang, W., Chen, E.: Multimodal human action recognition based on heterogeneous multistream networks. Signal Process. 36, 1422–1428 (2020)
Naeem, H.B., Yousaf, M.H., Khan, F.H., Yasin, A.: Vision-based real-time human malicious behavior detection. 2021 International Conference on Artificial Intelligence (ICAI), 193–197 (2021). https://doi.org/10.1109/ICAI52203.2021.9445205
Nam, H., Han, B.: Learning multi-domain convolutional neural networks for visual tracking. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4293–4302 (2016)
Panpan, X., Yun, L., Hui, L., et al.: Human behavior recognition based on time-domain expanded residual network and two-branch structure. Control Decision Making 37, 2993–3002 (2022)
Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3d residual networks. 2017 IEEE International Conference on Computer Vision (ICCV), 5534–5542 (2017)
Voigtlaender, P., Luiten, J., Torr, P.H.S., Leibe, B.: Siam r-cnn: Visual tracking by re-detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6577–6587 (2020)
Wang, Z., Gao, B.: Abnormal behavior recognition based on spatio-temporal fusion convolutional neural network. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 41, 2052–2056 (2020)
Wang, H., Su, H., Long, G., Wang, Y., Yin, K.: Parking anomaly behavior recognition method based on key sentence of behavior sequence features. Computer Science 46(10), 299–306 (2019). https://doi.org/10.11896/jsjkx.180901750
Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.P.: Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In: ECCV, pp. 318–335 (2018)
Yuan, J., Wu, X., Yuan, S.: A rapid recognition method for pedestrian abnormal behavior. 2020 International Conference on Computer Vision, Image and Deep Learning (CVIDL), 241–245 (2020). https://doi.org/10.1109/CVIDL51233.2020.00-93
Zhao, J., Snoek, C.G.M.: Dance with flow: Two-in-one stream action detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9927–9936 (2019). https://doi.org/10.1109/CVPR.2019.01017
Acknowledgements
The authors would like to acknowledge the support from the National Natural Science Foundation of China (grant No. 61906125).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ji, X., Zhao, S. & Li, J. An algorithm for abnormal behavior recognition based on sharing human target tracking features. Int J Intell Robot Appl (2024). https://doi.org/10.1007/s41315-024-00329-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s41315-024-00329-y