Abstract
Group Activity Recognition is to recognize and classify different actions or activities appearing in the video. The detailed description of human actions and group activities is essential information, which can be used in real-time CCTV video surveillance, health care, sports video analysis, etc. The existing methods, such as pose estimation based and graph network based group activity recognition can perform reasonable group activity understanding, however those models have bad performance on video with extreme brightness and contrast condition. This study proposes an improved actor relation graph based model (IARG) that mainly focused on group activity recognition by learning the pair-wise actor appearance similarity and actor positions. We propose to use Normalized cross-correlation (NCC) and the sum of absolute differences (SAD) to calculate the pair-wise appearance similarity and build the actor relationship graph to allow the graph convolution network to learn how to classify group activities. We demonstrate that our approach significantly outperforms existing state-of-the-art techniques on the public group activity recognition datasets called collective activity dataset and augmented dataset. Visualized results (sample frames can be found in Appendix) can further demonstrate each input video frame with predicted bounding boxes on each human object and both predicted individual action and collective activities.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gao, L., Guo, Z., Zhang, H., Xu, X., Shen, H.T.: Video captioning with attention-based LSTM and semantic consistency. IEEE Trans. Multimedia 19(9), 2045–2055 (2017)
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., Saenko, K.: Translating videos to natural language using deep recurrent neural networks. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2015)
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., Saenko, K.: Sequence to sequence - video to text. In: 2015 IEEE International Conference on Computer Vision (ICCV) (2015)
Sherstinsky, A.: Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Phys. D Nonlinear Phenomena 404, 132306 (2020). https://www.sciencedirect.com/science/article/pii/S0167278919305974
Krishna, R. Hata, K., Ren, F., Fei-Fei, L., Niebles, J. C.: Dense-captioning events in videos. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)
Fernando, B., Chet, C.T.Y., Bilen, H.: Weakly supervised gaussian networks for action detection. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020)
Noori, F.M., Wallace, B., Uddin, M.Z., Torresen, J.: A robust human activity recognition approach using openpose, motion features, and deep recurrent neural network. In: Felsberg, M., Forssén, P.-E., Sintorn, I.-M., Unger, J. (eds.) SCIA 2019. LNCS, vol. 11482, pp. 299–310. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20205-7_25
Cao, Z., Martinez, G.H., Simon, T., Wei, S.-E., Sheikh, Y.A.: Openpose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. PP, 1 (2019)
Wu, J., Wang, L., Wang, L., Guo, J., Wu, G.: Learning actor relation graphs for group activity recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019
Ibrahim, M.S., Mori, G.: Hierarchical relational networks for group activity recognition and retrieval. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 742–758. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_44
Howard, A.G., et al: MobileNets: efficient convolutional neural networks for mobile vision applications (2017).https://arxiv.org/abs/1704.04861
Li, X., Zhou, Z., Chen, L., Gao, L.: Residual attention-based LSTM for video captioning. World Wide Web 22(2), 621–636 (2018). https://doi.org/10.1007/s11280-018-0531-z
Heath, C.D.C., Heath, T., McDaniel, T., Venkateswara, H., Panchanathan, S.: Using participatory design to create a user interface for analyzing pivotal response treatment video probes. In: McDaniel, T., Berretti, S., Curcio, I.D.D., Basu, A. (eds.) ICSM 2019. LNCS, vol. 12015, pp. 183–198. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-54407-2_16
Raaj, Y., Idrees, H., Hidalgo, G., Sheikh, Y.: Efficient online multi-person 2D pose tracking with recurrent spatio-temporal affinity fields. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Yuan, H., Ni, D., Wang, M.: Spatio-temporal dynamic inference network for group activity recognition. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
Li, et al.: GroupFormer: group activity recognition with clustered spatial-temporal transformer. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: IEEE International Conference on Computer Vision (ICCV), vol. 2017, pp. 2980–2988 (2017)
Raghavender Rao, Y.: Application of normalized cross correlation to image registration. Int. J. Res. Eng. Technol. 03(17), 12–16 (2014)
Choi, W., Shahid, K., Savarese, S.: What are they doing?: collective activity classification using spatio-temporal relationship among people. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops (2009)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision, pp. 2818-2826 (2015). https://arxiv.org/abs/1512.00567
Acknowledgment
The authors would like to thank our mentor Dr. Nasim Hajari, Postdoctoral Fellow, Department of Computing Science, University of Alberta, for her guidance and feedback throughout the research and study. We would also thank our advisor Dr. Anup Basu for their motivation and support to bring out the novelty in our research. Finally, We would like to thank the researchers of the previous work, which is the inspiration and starting point for our research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
Visualization of results on 6 test video clips
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kuang, Z., Tie, X. (2022). IARG: Improved Actor Relation Graph Based Group Activity Recognition. In: Berretti, S., Su, GM. (eds) Smart Multimedia. ICSM 2022. Lecture Notes in Computer Science, vol 13497. Springer, Cham. https://doi.org/10.1007/978-3-031-22061-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-22061-6_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22060-9
Online ISBN: 978-3-031-22061-6
eBook Packages: Computer ScienceComputer Science (R0)