Abstract
Videos are composed of multiple tasks. Dense video captioning entails captioning of different events in the video. A textual description is generated based on visual, speech and audio cues from a video and then topic modeling is performed on the generated caption. Uncertainty modeling technique is applied for finding temporal event proposals where timestamps for each event in the video are produced and also uses Transformer which inputs multi-modal features to identify captions effectively and to make it more precise. Topic modeling tasks include highlighted keywords in the captions generated and topic generation i.e., category under which the whole caption belongs to. The proposed model generates a textual description based on the dynamic and static visual features and audio cues from a video and then topic modeling is performed on the generated caption.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Xiong Y, Dai B, Lin D (2018) Move forward and tell: a progressive generator of video descriptions. In: Proceedings of the European conference on computer vision, pp 468–483
Chadha A, Arora G, Kaloty N (2022) iPerceive: applying common-sense reasoning to multi-modal dense video captioning and video question answering, pp 1–13
Hao W, Zhang Z, Guan H (2018) Integrating both visual and audio cues for enhanced video caption. In: Proceedings of the AAAI conference on artificial intelligence, vol 32, no 1
Buch S, Escorcia V, Shen C, Ghanem B, Niebles J (2017) SST: single-stream temporal action proposals. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 6373–6382
Rahman T, Xu B, Sigal L (2019) Watch, listen and tell: multi-modal weakly supervised dense event captioning. In: IEEE/CVF international conference on computer vision (ICCV), pp 8907–8916
Hori C, Hori T, Lee T, Zhang Z, Harsham B, Hershey J, Marks T, Sumi K (2017) Attention-based multimodal fusion for video description. In: Proceedings of the IEEE international conference on computer vision, pp 4193–4202
Li Y, Yao T, Pan Y, Chao H, Mei T (2018) Jointly localizing and describing events for dense video captioning. In: IEEE/CVF conference on computer vision and pattern recognition, pp 7492–7500
Zhao B, Li X, Lu X (2018) Video captioning with tube features. In: Proceedings of the 27th international joint conference on artificial intelligence (IJCAI'18). AAAI Press, pp 1177–1183
Baraldi L, Grana C, Cucchiara R (2017) Hierarchical boundary-aware neural encoder for video captioning. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 3185–3194
Deng C, Chen S, Chen D, He Y, Wu Q (2021) Sketch, ground, and refine: top-down dense video captioning. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 234–243
Carreira J, Zisserman A (2017) Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6299–6308
Gao L, Lei Y, Zeng P, Song J, Wang M, Shen H (2022) Hierarchical representation network with auxiliary tasks for video captioning and video question answering. IEEE Trans Image Process 31:202–215
Zhou L, Zhou Y, Corso J, Socher R, Xiong C (2018) End-to-end dense video captioning with masked transformer. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 8739–8748
Venugopalan S, Xu H, Donahue J, Rohrbach M, Mooney R, Saenko K (2015) Translating videos to natural language using deep recurrent neural networks. In: Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 1494–1504
Donahue J, Hendricks L, Rohrbach M, Venugopalan S, Guadarrama S, Saenko K, Darrell T (2015) Long-term recurrent convolutional networks for visual recognition and description, pp 2625–2634
Hershey S, Chaudhuri S, Ellis D, Gemmeke J, Jansen A, Moore C, Plakal M, Platt D, Saurous R, Seybold B (2017) CNN architectures for large-scale audio classification. In: 2017 IEEE international conference on acoustics, speech and signal process, pp 131–135
Lee S, Kim I (2018) Multimodal feature learning for video captioning. Math Probl Eng 1–8
ActivityNet dataset: http://activity-net.org/
Lee P, Wang J, Lu Y, Byun H (2021) Weakly-supervised temporal action localization by uncertainty modeling. In: The thirty-fifth AAAI conference on artificial intelligence, pp 1854–1862
Sumalatha MR, Lakshmi Harika P, Aravind J, Dhaarani S, Rajavi P (2018) Detection of static and dynamic abnormal activities in crowded areas using hybrid clustering. In: International conference on ISMAC in computational vision and bio-engineering, pp 815–824
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Koshti D, Kamoji S, Kalnad N, Sreekumar S, Bhujbal S (2020) Video anomaly detection using inflated 3D convolution network. In: International conference on inventive computation technologies (ICICT), pp 729–733
Iashin V, Rahtu E (2020) Multi-modal dense video captioning. In: IEEE/CVF conference on computer vision and pattern recognition (2020)
Pennington J, Socher R, Manning C (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language process (EMNLP), pp 1532–1543
Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network, vol 2, no 7. arXiv preprint arXiv:1503.02531
Grootendorst M (2020) Keybert: minimal keyword extraction with bert
Krishna R, Hata K, Ren F, Fei-Fei L, Niebles J (2017) Dense-captioning events in videos. In: 2017 IEEE international conference on computer vision (ICCV), pp 706–715
Zhang Z, Xu D, Ouyang W, Tan C (2020) Show, tell and summarize: dense video captioning using visual cue aided sentence summarization. IEEE Trans Circuits Syst Video Technol 30(9):3130–3139
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Palivela, L.H., Swetha, S., Nithish Guhan, M., Prasanna Venkatesh, M. (2023). Dense Video Captioning Using Video-Audio Features and Topic Modeling Based on Caption. In: Bindhu, V., Tavares, J.M.R.S., Vuppalapati, C. (eds) Proceedings of Fourth International Conference on Communication, Computing and Electronics Systems . Lecture Notes in Electrical Engineering, vol 977. Springer, Singapore. https://doi.org/10.1007/978-981-19-7753-4_40
Download citation
DOI: https://doi.org/10.1007/978-981-19-7753-4_40
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-7752-7
Online ISBN: 978-981-19-7753-4
eBook Packages: EngineeringEngineering (R0)