Enhanced video temporal segmentation using a Siamese network with multimodal features

Abstract

Shot boundary detection (SBD) is a critical pre-processing task for intelligent video analysis applications. In this study, we proposed a novel multimodal approach for SBD by utilizing a Siamese network with multimodal features to learn the distance measure between audiovisual features. To extract relevant features from the audio stream’s power spectrum density (PSD), we combined Tsfresh, a Python package for time series feature extraction, with PCA (principal component analysis), and used a Gru-Attention network for learning sequential semantic representations and spatial location information. For the visual modality, we employed the pre-trained EfficientNet model to extract and learn visual features. Our proposed network learned the similarity score from the image embedding features and the PSD as audio features, which were then used to build a signal representing the audiovisual change. We used a global threshold for transition detection and an adaptive threshold to differentiate between the detected transition types (abrupt or gradual). In our experimental study, we applied the proposed approach to standard datasets (TRECvid 2001 and TRECvid 2007) and found that the introduction of audio features achieved a significant improvement in terms of F1 score (92.43%) and gradual transition detection (90.08%) compared to state-of-the-art models.