Abstract
Temporal action detection is used to detect the start and end times and classify the potentially specific actions in a video. Prior studies in temporal action detection perform weak because they can not fully understand the whole input video's temporal structure and context information, and fail to adapt to the diversity of action time span. We propose a novel Timeception Single Shot Action Detector (TC-SSAD) to solve the problems mentioned above. In detail, we leverage the multiple Timeception layers to generate multi-scale feature sequences, where each Timeception layer uses depthwise-separable temporal convolution with multi-scale convolution kernels to capture the diversity of time spans. Besides, we use the super-event modules to learn the entire input video’s temporal structure and contextual information. The experimental results on THUMOS14 dataset show that when IoU threshold is 0.5, our method achieves 38.2% and 44.3% mAP on Two-stream features and Two-stream i3D features respectively, which is better than Decouple-SSAD network based method by 2.4% and 0.6%. Our method on Activitynet-1.3 dataset achieves 20.4% mAP, which is better than Decouple-SSAD network based method by 0.61% as far as Two-stream features on concerned.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ren, J., Yuan, L., Nurmi, P., et al.: Camel: Smart, adaptive energy optimization for mobile web interactions. In: Proceedings of the IEEE INFOCOM Conference on Computer Communications, pp. 119–128 (2020)
Qin, Q., Ren, J., Yu, J., et al.: To compress, or not to compress: characterizing deep learning model compression for embedded inference. In: 2018 IEEE International Conference on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom), pp. 729–736 (2018)
Y.-W., Chao, S., Vijayanarasimhan, B., et al.: Rethinking the faster R-CNN architecture for temporal action localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1130–1139 (2018)
Wang, J., Jiang, W., Ma, L., et al.: Bidirectional attentive fusion with context gating for dense video captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7190–7198 (2018)
Huang, Y., Dai, Q., Lu, Y.: Decoupling localization and classification in single shot temporal action detection. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 1288–1293 (2019)
Shou, Z., Wang, D., Chang, S.F.: Temporal action localization in untrimmed videos via multi-stage cnns. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1049–1058 (2016)
Tran, D., Bourdev, L., Fergus, R., et al.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp. 4489–4497 (2015)
Gao, J., Yang, Z., Chen, K., et al.: Turn tap: temporal unit regression network for temporal action proposals. In: Proceedings of the IEEE international conference on computer vision, pp. 3628–3636 (2017)
Xiong, Y., Zhao, Y., Wang, L., et al.: A pursuit of temporal accuracy in general activity detection. arXiv preprint arXiv:1703.02716 (2017)
Zhao, Y., Xiong, Y., Wang, L., et al.: Temporal action detection with structured segment networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2914–2923 (2017)
Lin, T., Zhao, X., Su, H., Wang, C., Yang, M.: BSN: boundary sensitive network for temporal action proposal generation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 3–21. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_1
Shou, Z., Chan, J., Zareian, A., et al.: CDC: convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5734–5743 (2017)
Ren, S., He, K., Girshick, R., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2016)
Xu, H., Das, A., Saenko, K.: R-c3d: region convolutional 3D network for temporal activity detection. In: Proceedings of the IEEE international conference on computer vision, pp. 5783–5792 (2017)
Idrees, H., Zamir, A.R., Jiang, Y.G., et al.: The THUMOS challenge on action recognition for videos “in the wild”. Comput. Vis. Image Understand. 155, 1–23 (2017)
Lin, T., Zhao, X., Shou, Z.: Single shot temporal action detection. In: Proceedings of the 25th ACM international conference on Multimedia, pp. 988–996 (2017)
Buch, S., Escorcia, V., Ghanem, B., et al.: End-to-end, single-stream temporal action detection in untrimmed videos. In: Procedings of the British Machine Vision Conference 2017. British Machine Vision Association, pp. 1–12 (2017)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Hussein, N., Gavves, E., Smeulders, A.W.M.: Timeception for complex action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 254–263 (2019)
Piergiovanni, A.J., Ryoo, M.S.: Learning latent super-events to detect multiple activities in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5304–5313 (2018)
Heilbron, F.C., Escorcia, V., Ghanem, B., et al.: Activitynet: a large-scale video benchmark for human activity understanding. In: Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961–970 (2015)
Singh, B., Marks, T.K., Jones, M., et al.: A multi-stream bi-directional recurrent neural network for fine-grained action detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1961–1970 (2016)
Long, F., Yao, T., Qiu, Z., et al.: Gaussian temporal awareness networks for action localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 344–353 (2019)
Buch, S., Escorcia, V., Shen, C., et al.: Sst: single-stream temporal action proposals. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2911–2920 (2017)
Zeng, R., Huang, W., Tan, M., et al.: Graph convolutional networks for temporal action localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7094–7103 (2019)
Yuan, Z., Stroud, J.C., Lu, T., et al.: Temporal action localization by structured maximal sums. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3684–3692 (2017)
Acknowledgements
This work was supported by National Natural Science Foundation of China (61877038, 61501287, 61902229) and Fundamental Research Funds for the Central Universities (No. TD2020044Y, No. GK201703058, No. GK202103084).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, X., Ma, M., Tian, Z., Ren, J. (2021). Timeception Single Shot Action Detector: A Single-Stage Method for Temporal Action Detection. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12888. Springer, Cham. https://doi.org/10.1007/978-3-030-87355-4_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-87355-4_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87354-7
Online ISBN: 978-3-030-87355-4
eBook Packages: Computer ScienceComputer Science (R0)