AHMN: A multi-modal network for long MOOC videos chapter segmentation

Abstract

This paper proposes a task named MOOC Videos Chapter Segmentation (MVCS) which is a significant problem in the field of video understanding. To solve this problem, we first introduce a dataset called MOOC Videos Understanding (MVU) which consists of approximately 10k annotated chapters organized by 120k snippets from 400 MOOC videos where chapters and snippets are two levels of video unit proposed in this paper for hierarchical level expression of videos. Then, we design the Attention-based Hierarchical bi-LSTM Multi-modal Network (AHMN) based on three core ideas: (1) we take advantage of the features of multi-modal semantic elements, including video, audio, and text, along with an attention-based multi-modal fusion module to extract video information in a comprehensive way. (2) we focus on chapters boundaries rather than the content recognition of chapters themselves, so we develop Boundary Predict Network (BPN) to label boundaries between chapters. (3) we exploit the semantic consistency between snippets and develop Consistency Modeling as an auxiliary task to improve the performance of BPN. Our experiments demonstrate that the proposed AHMN can solve the MVCS precisely, outperforming previous methods on all evaluation metrics.

Appendices

Appendix A    Appendices

1.1 A.1   APPENDIX: Dataset Collection

The MOOC Videos Understanding Dataset (MVU) is introduced in Section 3 of the paper. Here we introduce additional details about the data source. In order to build a reliable and easily scalable MOOC videos dataset, we use selenium³ to collect a total of 400 videos from 20 categories from the videolectures⁴. In the videolectures, usually a webpage contains the following elements: a video player, a slide player, a timeline where you can click to select any slide page and jump to its corresponding video clip. The videolectures regularly uploads the latest MOOC videos, so the data source is reliable and the fully automated collection pipline also facilitates the easy expansion of the MVU.

Fig. 6

Overview of our text network. Our text network consists of two parts, the LSTM-based network on the left is snippet’s word2vec sentence vector $\partial _{i}^{w2v}$, the right side is a BERT-based sentence vector $\partial _{i}^{BERT}$

1.2 A.2   APPENDIX: Text Network

The text network is introduced in Section 4.1.1 in this paper. We propose a two-stage feature extraction framework for text information as shown in Fig. 6. Here we introduce more implementation details on this framework. Inspired by [43], we develop a Bi-directional Long Short-Term Memory (bi-LSTM) network to encode the words. For each snippet, the input to the network is the sequence of words $[w_{1}^{i},w_{2}^{i},\cdot \cdot \cdot ,w_{k}^{i}]$ and the output is the sentence representation $\partial _{i}^{w2v}$ which is from bi-LSTM output layer after maxpooling, with a dimension of 512. Then to further enhance the feature representation of text modality, we use the BERT [41] pre-trained model, for each text sentence, inserting [CLS] at the beginning and [SEP] at the end, and feeding the sentence with the marking symbols into the BERT. A 768-dimensional vector of [CLS] for each sentence in the results, projected into a 512-dimensional vector by a linear layer, represents the BERT feature $\partial _{i}^{BERT}$ of the snippet.

Additional qualiative results

Fig. 7

The role of multi-modal features. The cosine similarity values of each semantic element is represented by the corresponding bar length

The effectiveness of multi-modal semantic information is shown in Section 5.3 in this paper, where we select typical MOOC video examples to illustrate how different modalities help the AHMN group several semantically similar snippets into one chapter and predict the chapter boundaries. To help further understand the role of multi-modal features, we present more visualization results of the AHMN on the MVCS task which are shown in Fig. 7. As is shown in Fig. 7(a), the speaker is chatting with his audience on this slide, so the text from both the speaker and the audience are not similar enough among these three snippets where the highly similar video features help the AHMN to confirm the boundary. Figure 7(b) shows a typical lecture scene, the camera moves a bit, resulting in the video features change a lot among these snippets, so the AHMN determines this chapter boundary based on similar text and audio features. Similar to Fig. 7(b), snippets in Fig. 7(c) do not have highly similar video features, but due to the noisy audience discussion, the AHMN could only determine the boundaries based on highly similar text features. Figure 7(d) shows a large-scale public class scene, where the camera moves a lot between the speakers and the screen, so the AHMN uses text information and audio information from the speaker. The speaker in Fig. 7(e) is playing a video clip with music in which there is not plenty of similar text information, so the AHMN uses the audio features of this video clip to recognize these snippets belong to the same chapter.