Abstract
Temporal language queries grounding in video aims to retrieve one specific moment in a long, untrimmed video by a query sentence. This is a challenging issue as a video may contain multiple moments of interests which have complex temporal dependencies with other temporal moments. To preserve the original video moment information during the multiple layers convolution operations, this paper introduces residual learning into the issue and proposes a novel semantic modulation based residual network (SMRN) that incorporates dynamical semantic modulation and multiple prediction maps in a single-shot feed-forward framework. Semantic modulation mechanism dynamically modulates the residual learning by assigning where to pay the visual attention. And the mechanism is able to adjust the weights given to different video moments with the guide of query sentence. Additionally, the network combines multiple feature maps from different layers to naturally captures different temporal relationships for precisely matching video moment and sentence. We evaluate our model on three datasets, i.e., TACoS, Charades-STA, and ActivityNet Caption, with significant improvement over the current state-of-the-arts. Furthermore ablation experiments were performed to show the effectiveness of our model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Krishna, R., Hata, K., Ren, F., Li, F.F.: Dense-captioning events in videos. In: Proceedings of International Conference on Computer Vision, pp. 706–715. IEEE, Venice (2017)
Gao, J., Sun, C, Yang, Z., Nevatia, R.: TALL: temporal activity localization via language query. In: Proceedings of International Conference on Computer Vision, pp. 5267–5275. IEEE, Venice (2017)
Hendricks, L.A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., Russell, B.: Localizing moments in video with temporal language. In: Proceedings of International Conference on Computer Vision, pp. 5803–5812. IEEE, Venice (2017)
Ge, R., Gao, J., Chen, K., Nevatia, R.: MAC: mining activity concepts for language-based temporal localization. In: Proceedings of the Winter Conference on Applications of Computer Vision, pp. 245–253. IEEE, Waikoloa (2019)
Zhang, S., Peng, H., Fu, J., Luo, J.: Learning 2D temporal adjacent networks for moment localization with natural language. In: proceedings of AAAI Conference on Artificial Intelligence. AAAI, New York (2020)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Manohar, P.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4489–4497. IEEE, Boston (2015)
Liu, M., Wang, X., Nie, L., He, X., Chen, B., Chua, T.: Attentive moment retrieval in videos. In: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 15–24. ACM, Ann Arbor (2018)
Wang, W., Huang, Y., Wang, L.: Language driven temporal activity localization: a semantic matching reinforcement learning model. In: Proceedings of Computer Vision and Pattern Recognition, pp. 334–343. IEEE, California (2019)
Hahn, M., Kadav, A., Rehg, J.M., Graf, H.P.: Tripping through time: efficient localization of activities in videos. arXiv preprint arXiv:1904.09936
Wang, J.W., Ma, L., Jiang, W.H.: Temporally grounding language queries in videos by contextual boundary-aware prediction. In: Proceedings of AAAI Conference on Artificial Intelligence. AAAI, New York (2020)
Chen, J., Chen, X., Ma, L., Jie, Z., Chua, T.S.: Temporally grounding natural sentence in video. In: Proceedings of Conference on Empirical Methods in Natural Language Processing, pp. 162–171. ACL, Belgium (2018)
Zhang, Z., Lin, Z., Zhao, Z., Xiao, Z.: Cross-modal interaction networks for query-based moment retrieval in videos. In: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 655-664. ACM, Paris (2019)
Zhang, D., Dai, X., Wang, X., Wang, Y.F., Davis, L.S.: MAN: moment alignment network for natural language moment retrieval via iterative graph adjustment. In: Proceedings of Computer Vision and Pattern Recognition, pp. 1247–1257. IEEE, California (2019)
Xu, H., He, K., Plummer, B.A., Sigal, L., Sclaroff, S., Saenko, K.: Multilevel language and vision integration for text-to-clip retrieval. In: Proceedings of AAAI Conference on Artificial Intelligence, pp. 9062–9069 AAAI, Arlington (2019)
Liu, M., Wang, X., Nie, L., Tian, Q., Chen, B., Chua, T.S.: Cross-modal moment localization in videos. In: Proceedings of ACM International Conference on Multimedia, pp. 843–851. ACM, Seoul (2018)
Yuan, Y., Mei, T., Zhu, W.: To find where you talk: temporal sentence localization in video with attention based location regression. In: Proceedings of AAAI Conference on Artificial Intelligence, pp. 9159–9166. AAAI, Arlington (2019)
Chen, S., Jiang, Y.G.: Semantic proposal for activity localization in videos via sentence query. In: Proceedings of AAAI Conference on Artificial Intelligence, pp. 8199–8206. AAAI, Arlington (2019)
Acknowledgements
This work was supported in part by National Natural Science Foundation of China under grant 61771145 and 61371148.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, C., Gu, X. (2020). Semantic Modulation Based Residual Network for Temporal Language Queries Grounding in Video. In: Han, M., Qin, S., Zhang, N. (eds) Advances in Neural Networks – ISNN 2020. ISNN 2020. Lecture Notes in Computer Science(), vol 12557. Springer, Cham. https://doi.org/10.1007/978-3-030-64221-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-64221-1_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64220-4
Online ISBN: 978-3-030-64221-1
eBook Packages: Computer ScienceComputer Science (R0)