Abstract
The present weakly-supervised methods for Temporal Action Localization are primarily responsible for capturing the temporal context. However, these approaches have limitations in capturing semantic context, resulting in the risk of ignoring snippets that are far apart but sharing the same action categories. To address this issue, we propose an action label propagation network utilizing sparse graph networks to effectively explore both temporal and semantic information in videos. The proposed SGLP-Net comprises two key components. One is the multi-scale temporal feature embedding module, a novel method that extracts both local and global temporal features of the videos during the initial stage using CNN and self-attention and serves as a generic module. The other is an action label propagation mechanism, which uses graph networks for feature aggregation and label propagation. To avoid the issue of excessive feature completeness, we optimize training using sparse graph convolutions. Extensive experiments are conducted on THUMOS14 and ActivityNet1.3 benchmarks, among which advanced results demonstrate the superiority of the proposed method. Code can be found at https://github.com/xyao-wu/SGLP-Net.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4733 (2017). https://doi.org/10.1109/CVPR.2017.502
Chao, Y.W., Vijayanarasimhan, S., Seybold, B., Ross, D.A., Deng, J., Sukthankar, R.: Rethinking the faster R-CNN architecture for temporal action localization. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1130–1139 (2018). https://doi.org/10.1109/CVPR.2018.00124
Chen, J., Ma, T., Xiao, C.: FastGCN: fast learning with graph convolutional networks via importance sampling. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=rytstxWAW
Chen, M., Gao, J., Yang, S., Xu, C.: Dual-evidential learning for weakly-supervised temporal action localization. In: Avidan, S., Brostow, G., Cissa, M., Farinella, G.M., Hassner, T. (eds.) European Conference on Computer Vision, vol. 13664, pp. 192–208. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19772-7_12
Chen, T., Li, B., Tao, Y., Wang, Y., Zhu, Y.: Class-incremental learning with multiscale distillation for weakly supervised temporal action localization. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds.) ICONIP 2022. LNCS, vol. 13623, pp. 367–378. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-19772-7_12
Ciptadi, A., Goodwin, M.S., Rehg, J.M.: Movement pattern histogram for action recognition and retrieval. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 695–710. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_45
Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89(1), 31–71 (1997). https://doi.org/10.1016/S0004-3702(96)00034-3. https://www.sciencedirect.com/science/article/pii/S0004370296000343
Douze, M., Szlam, A., Hariharan, B., Jégou, H.: Low-shot learning with large-scale diffusion. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3349–3358 (2018). https://doi.org/10.1109/CVPR.2018.00353
He, B., Yang, X., Kang, L., Cheng, Z., Zhou, X., Shrivastava, A.: ASM-Loc: action-aware segment modeling for weakly-supervised temporal action localization. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13915–13925 (2022). https://doi.org/10.1109/CVPR52688.2022.01355
Heilbron, F.C., Escorcia, V., Ghanem, B., Niebles, J.C.: Activitynet: a large-scale video benchmark for human activity understanding. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961–970 (2015). https://doi.org/10.1109/CVPR.2015.7298698
Idrees, H., et al.: The thumos challenge on action recognition for videos “in the wild”. Comput. Vision Image Underst. 155, 1–23 (2017). https://doi.org/10.1016/j.cviu.2016.10.018. https://www.sciencedirect.com/science/article/pii/S1077314216301710
Kay, W., et al.: The kinetics human action video dataset (2017)
Lee, P., Uh, Y., Byun, H.: Background suppression network for weakly-supervised temporal action localization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 11320–11327 (2020). https://doi.org/10.1609/aaai.v34i07.6793
Li, L., Kong, T., Sun, F., Liu, H.: Deep point-wise prediction for action temporal proposal. In: Gedeon, T., Wong, K.W., Lee, M. (eds.) ICONIP 2019. LNCS, vol. 11955, pp. 475–487. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-36718-3_40
Liu, D., Jiang, T., Wang, Y.: Completeness modeling and context separation for weakly supervised temporal action localization. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1298–1307 (2019). https://doi.org/10.1109/CVPR.2019.00139
Luo, W., et al.: Action unit memory network for weakly supervised temporal action localization. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9964–9974 (2021). https://doi.org/10.1109/CVPR46437.2021.00984
Moniruzzaman, M., Yin, Z.: Feature weakening, contextualization, and discrimination for weakly supervised temporal action localization. IEEE Trans. Multimedia, 1–13 (2023). https://doi.org/10.1109/TMM.2023.3263965
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML 2010, pp. 807–814. Omnipress, Madison (2010)
Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization by sparse temporal pooling network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017). https://doi.org/10.1109/CVPR.2018.00706
Qu, S., Chen, G., Li, Z., Zhang, L., Lu, F., Knoll, A.: Acm-net: action context modeling network for weakly-supervised temporal action localization. arXiv preprint arXiv:2104.02967 (2021)
Raghavan, U.N., Albert, R., Kumara, S.: Near linear time algorithm to detect community structures in large-scale networks. Phys. Rev. E 76, 036106 (2007). https://doi.org/10.1103/PhysRevE.76.036106
Shi, B., Dai, Q., Mu, Y., Wang, J.: Weakly-supervised action localization by generative attention modeling. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1006–1016 (2020). https://doi.org/10.1109/CVPR42600.2020.00109
Shi, D., Zhong, Y., Cao, Q., Ma, L., Li, J., Tao, D.: Tridet: temporal action detection with relative boundary modeling. arXiv:2303.07347 (2023)
Su, H., Zhao, X., Lin, T., Fei, H.: Weakly supervised temporal action detection with shot-based temporal pooling network. In: Cheng, L., Leung, A.C.S., Ozawa, S. (eds.) ICONIP 2018. LNCS, vol. 11304, pp. 426–436. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04212-7_37
Yang, W., Zhang, T., Yu, X., Qi, T., Zhang, Y., Wu, F.: Uncertainty guided collaborative training for weakly supervised temporal action detection. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 53–63 (2021). https://doi.org/10.1109/CVPR46437.2021.00012
Yang, Z., Qin, J., Huang, D.: ACGNet: action complement graph network for weakly-supervised temporal action localization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 3090–3098 (2022)
Zeng, R., et al.: Graph convolutional networks for temporal action localization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7093–7102 (2019). https://doi.org/10.1109/ICCV.2019.00719
Zhai, Y., Wang, L., Tang, W., Zhang, Q., Yuan, J., Hua, G.: Two-stream consensus network for weakly-supervised temporal action localization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 37–54. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_3
Zhang, C.L., Wu, J., Li, Y.: ActionFormer: localizing moments of actions with transformers. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13664, pp. 492–510. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19772-7_29
Zhao, C., Thabet, A., Ghanem, B.: Video self-stitching graph network for temporal action localization. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13638–13647 (2021). https://doi.org/10.1109/ICCV48922.2021.01340
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wu, X., Song, Y. (2024). SGLP-Net: Sparse Graph Label Propagation Network for Weakly-Supervised Temporal Action Localization. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14451. Springer, Singapore. https://doi.org/10.1007/978-981-99-8073-4_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-8073-4_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8072-7
Online ISBN: 978-981-99-8073-4
eBook Packages: Computer ScienceComputer Science (R0)