1 Introduction

Movie analysis is a method which takes into account the analyzing and discovery movie’s content. The goal of this work is to serve the easier method for movie understanding. Normally, when audiences watch the given movie, they usually discover the plotline and the characters’ relationships. In this regard, the main characters (“the protagonist” or “the hero”) are started to be recognized. Besides, the protagonist makes audience experience the story, and other characters support the protagonist to tell the story of the movie. The characters can be extracted to analyze the relationships among them. Figure 1 introduces the archetypes and the roles of characters in the movie.

Movie analysis research takes into account extracting summarization version where the goal is to summarize a movie to provide an effective shortened version of the film in a condensed and succinct representation of the storyline. This task works with a combination of analysis and processing for movie frames, shots, scenes, segmentation and textual description. The simplest summarization method considers directly analyzing a sample from a movie, including keyframes, shots and scenes during the movie. Such methods can be prepared according to different types of content that are used for video analysis with categories that include cognitive [1] and non-cognitive approaches [2]. Cognitive approaches extract features of the movie including auditory, visual and textual features to identify the importance of keyframes or scenes in the movie. Some of the proposed methods use low-level visual including audio, text-based, visual, audio–textual, audiovisual, and other features. In addition, hybrid methods such as [3] consider video/movie segmentation using scene detection and classification. Other studies involve extracting and analyzing events in the given movie. Non-cognitive methods extract video/movie summarization versions using user feedback [4]. In this regard, such concepts are considered to detect and recognize users’ emotion (e.g. [5] and [6]). These approaches aim to help the user to extract a shortened version by removing unimportant or redundant content in the movie. The proposed studies normally used movie/drama theory to analyze plotlines and to summarize a movie by focusing on the characters and the relationships among them. In addition, many effective methods have been proposed to explore the story of the movie using characters’ co-appearances [6] and characters’ dialogs [5].

Fig. 1
figure 1

The archetypes of characters in Episode IV of the Star War series

Full size image

According to movie theory, “All films are about nothing - nothing but characters” (e.g. [7] and [8]). Characters will, therefore, be detected, identified and recognized to discover the storyline of the movie through the use of a social network analysis strategy. The appearances and co-appearances of the characters have become the most important roles to extract the summarization version of the given movie. Therefore, social network techniques are used to analyze and discover the importance of the characters and to classify their roles in the movie (e.g. [9] and [10]). Performing the proposed methods involves a combination to benefit from effective techniques to ensure a correct rate of feature detection and recognition and effectiveness in the social network analysis. Besides, Shan et al. [11] provided a method to summarize the given movie using screen comments. In addition, Li et al. provided a method for assisting movie summarization using plot information [12]. Moreover, in the study of Mademlis et al., the shots are used for extracting movie summarization using semantic selection method [13]. In another approach, research in movie summarization focused on the metadata to achieve the algorithm [14]. The study of Sun et.al used the bullet screen comment to extract movie summarization [11]. Recent researches showed that among method have been introduced using story-based and content-based analysis. These methods have focused on the role of character, extracting the social network and analyzing it to discover the story of the given movie. However, the accuracy rate and processing time of these systems are quite unsatisfactory.

In this study, we introduce a method to index characters and extract a shortened version of a movie using character’s co-appearances. The proposed method considers the appearance of the characters and the relationships among them during the movie playbacks. Social network analysis can be used to find the importance during movie segmentation to summarize and index the movie. The difference with respect to existing methods (e.g. [9, 10, 15] and [16]) is that ours considers the appearances and co-appearances of characters to analyze the main storyline of the movie through social network techniques. Social network analysis and evaluation are then used to discover the storyline of the movie to help the audience gain a better understanding of the movie.

For the proposed method, we assume that the appearances of the characters are “on-screen”. Besides, the relationship among them with respect to the measurement and evaluation of centralities are processed. In addition, the social network is then used to classify the characters in certain communities, to discover the protagonist and to process the main storyline of the given movie. Moreover, the appearance of the characters in the movie is also used to measure and segment the importance. The analysis of the network of characters and the segmented scenes are then used to extract a shortened version of the movie.

Regarding movie study, visual abstractions can be helpful and effective in a variety of contexts, such as for multimedia archives, movie marketing and home entertainment. In multimedia archives, where a movie is stored, movie abstract should be useful for indexing and retrieval. For example, the internet movie database (IMDb) on the web [17] is indexed through “hand-made” textual information about movies. A short clip that is selected at random is also sometimes included. The index should be easily extended by automatically generating a shortened movie version [18].

Many existing research methods have been proposed including scene-based methods which consider user preferences and the causal relationships of the role of communities. The proposed method characterized the role of the interaction of the community via the role that community networks play to identify the scenes in the movie through human face clustering to detect and cluster faces of the characters. During the summarization stage, the proposed method summarizes the movie as a social network pruning problem. However, the limitations of the proposed method can be addressed by improving the accuracy of face recognition and the speed of detection and recognition. In addition, some important scenes may be crowded out from a summary version during pruning [10]. In another approach, the co-occurrence of the characters and the script are used to summarize the movie. This method involves a character story based on flow graph that can be used to conduct sub-story discovery in the movie by analyzing video segments with respect to the character histogram [9]. However, this method still has some limitations for character identification due to the lack of accuracy in the representation of the character histogram.

Regarding our study, we annotate the characters’ appearances during movie playback. In this regard, data of characters’ appearance are then used in the second step to analyze and discover the movie story line based on social network analysis techniques. Then, the appearances of characters are analyzed to identify the important part so that a shortened version can be produced. The main contribution of this research includes (1) a construction of the social network of the characters to determine the protagonist, character’s role and the storyline in the given movie, and (2) a method to formulate the movie summary and indexing using social network technique.

The organization of this paper is as follows. The main definition and analysis methods to summarize and index the movie will be described in Sect. 2. The summarization process is introduced in Sect. 3. The experimental evaluation will be shown in Sect. 4. The conclusion will be described in Sect. 5.

Fig. 2
figure 2

The framework of proposed method

Full size image

2 The appearances of characters

2.1 Character’s appearance model

To represent the character, we assume that he/she appears “on-screen” visually as mentioned in our previous study [19]. A character in a multimedia content has a time distribution. We can represent the character’s playing time distribution by means of a sequence of time intervals as the following.

Let \(C=\{ c_1, c_2,\ldots ,c_k \}\) be the set of characters. L is the total length that is calculated by a timestamp value. Time intervals is represented by \([t_1,t_2]\) where \(t_1\) and \(t_2\) are timestamps, and \(t_2>t_1\).

Definition 1

(The appearance of a character) Let \(c_i \in C\) be a character. The occurrences of \(c_i\) is represented as

$$\begin{aligned} T(c_i) = \left\langle \left[ t^{i-}_{1}, t^{i+}_{1}\right] , \ldots , \left[ t^{i-}_{n}, t^{i+}_{n}\right] \right\rangle \end{aligned}$$
(1)

where \(c_i \in C\); \(t^{i+}_{j} > t^{i-}_{j}\) for \(i=1,\ldots ,k; \; j=1,\ldots ,n\); \(t^{i}_{j} \in [0\ldots L]\); in a time point not belonging to intervals \(t^{i}_{1}, \ldots , t^{i}_{n}\) character \(c_i\) does not appear.

Using Definition 1, we measure the total time of characters that they appear together as the following.

Definition 2

(Total appearance time of characters) Total appearance time of characters \(\alpha _{ij}\) is the total of intervals in \(T(c_i\wedge c_j)\), where \(T(c_i\wedge c_j)\) is character’s appearance of \(c_i\) and \(c_j\).

$$\begin{aligned} \alpha _{ij} = |T(c_i\wedge c_j)| \end{aligned}$$
(2)

where \(T(c_i\wedge c_j)\) is character’s appearance of \(c_i\) and \(c_j\).

Besides, we also represent the number of appearances time between characters as follows

Definition 3

(Number of appearances time of characters) Number of appearances time of characters \(\beta _{ij}\) is the number of intervals in \(T(c_i\wedge c_j)\), where \(T(c_i\wedge c_j)\) is character’s appearances of \(c_i\) and \(c_j\).

We can note that \(\alpha _{ij = 0}\) iff \( \beta _{ij}=0\)

3 Character network analysis and movie summarization

3.1 Character network

In this work, we introduce a character network (CN—co-appearance network) to determine and analyze the importance of a characters in the given movie. In this regard, we construct the CN using the annotation data of the appearances and the co-appearances of the characters. CN is represented as the following weighted graph:

$$\begin{aligned} G = \langle {C, E} \rangle \end{aligned}$$
(3)

where \(C = \{c _1, c _2, \ldots , c _n\} \) represents the set of characters, \(E = \{<c_i,c_j,\alpha _{ij},\beta _{ij}>\}\) represents the relationships of characters.

The important task in movie study is to analyze and determine the protagonist. Using CN, we can discover the central node using the Eigenvector, closeness centralities and the Weighted degree evaluation. The combination of these values will be used to determine how the character importance is.

The main character (the protagonist) is discovered and characters are classified into certain communities using the score of the characters that have appeared by combining the closeness, the Eigenvector centrality and the weighted degree as the following:

$$\begin{aligned} S_{C_i}= E(c_i) * C(c_i) *\Omega (c_i) \end{aligned}$$
(4)

where \(c_i\) is the character i; \(S_{C_i}\) is the Score of the character \(c_i\), \(E(c_i)\) is the Eigenvector, \(C(c_i)\) is the closeness centrality, \(\Omega _i\) is the weighed degree and \(E(c_i)\), \(C(c_i)\), \(\Omega (c_i) \ne 0\).

The average value of the Eigenvector, closeness centrality and weighted degree could be measured as the following.

The average of the centralities is measured using

$$\begin{aligned} \varGamma _S = \frac{\sum (S_{C_i})}{n} \ \end{aligned}$$
(5)

where \(S_{C_i}\) is the score of the characters based on the closeness, Eigenvector centrality and weighted degree, n is the number of characters that are used for the measurement.

During the analysis stage, we construct a CN using the social network strategy that belongs to the CN. The nodes represent the character, the power of nodes represents the importance of the character, and the value of the edge represents the strength of the relationships among the characters. The results of the analysis are used to segment the characters into the main character class which consists of characters who play leading roles and minor characters class which are characters who play a supporting role in the given movie. The average of the centralities (as measured with Eq. 5) is used to segment the characters in certain communities. If the centrality of the characters is greater than the average score \(\varGamma _S\), then these characters belong to the main characters class. If the centrality of the characters is less than the average score \(\varGamma _S\), then these characters will belong to the minor character class.

3.2 Movie summarization

The summarization is intended to convey as much information as possible with a shorter movie clip and to present the audience the fastest means with which to understand the movie. In our research, we first detect the movie storyline according to the characters’ appearances. We then use the social network analysis technique to determine and discover the main story of the movie. Figure 2 illustrates the framework of the proposed method that is used to summarize the movie.

To extract summarization versions, we assume that the appearances of protagonist and characters who have the high value of centralities are the most important factors. In this regard, we extract two summarization versions of a movie as the following strategies.

  • Version 1 The appearances of the protagonist are selected.

  • Version 2 The appearances of the protagonist and main characters are selected.

Algorithm 1 illustrates the algorithm for extracting a version (version 1) of summarization by selecting the appearance of the protagonist of the given movie.

figure a

Besides, we proposed another algorithm to extract another version (version 2) of summarization by selecting the appearance distribution of the protagonist and the main characters as the following.

figure b

4 Results and experiments

We selected a total of 17 episodes from the Star Wars series MV: 1, 2, 3, 4, 5, 6, The Lord of Rings series MV: 7, 8, 9 and the Harry Potter series MV: 10, 11, 12, 13, 14, 15, 16, 17 to index and extract the summarization versions. The characters in these movies were annotated according to the time that they appeared and disappeared during movie play time. The movie was summarized by extracting the CN and performing a technique that takes into account the use of centrality in analyzing it. By the use of the Eigenvector, closeness centrality and weighted degree, CN will be analyzed to discover the protagonist, main characters and so on. The summarization and indexing system were developed in Java with the VLCj API [20] and Gephi API [21].

For representing the appearances of characters, in our previous study [22], we proposed a system that could assist user to index the appearances of characters as the following.

  • Parsing character list from IMDb database. This module allows user to fetch character list from IMDb database for each movie.

  • Character appearance annotation. This module allows the user to annotate the appearances of characters in the given movie by the order of character name, start time and end time of appearance. Data of this system will be stored in XML file.

  • Character network visualization: this module extracts and visualizes character network using Prefuse API.

Table 1 The centralities measurement of the Star Wars VI
Full size table

4.1 Experimental results

Table 1 and Figs. 3 and 4 illustrate the centrality of the characters in the MV4. Using closeness centrality, betweenness centrality and weighted degree measurement strategies for the total appearance of a character (Definition 1) and the total appearance time of character (Definition 2). These results show that the protagonist of this movie has the highest value of centrality and he/she holds the important roles which show that Luke Skywalker had the highest values. Comparing to IMDb database [17], Luke Skywalker is the protagonist who plays an important role in telling the main story.

Fig. 3
figure 3

The centralities measurement results using total appearance time of characters

Full size image
Fig. 4
figure 4

The centralities measurement results using number of appearances time of characters

Full size image
Fig. 5
figure 5

Main characters’ precision and recall

Full size image

Figures 5 and 6 illustrate the accuracy of our proposed method. We use 17 movies for the experimental evaluation. These results showed that the main class exhibited 87% of precision and 83% of recall on average. The minor class exhibited 88% of precision and 83% of recall on average. In another approach, the number of character co-appearances is counted, and the main class accomplished 82% of precision and 83% of recall on average and the minor class achieves 82% of precision and 81% of recall on average.

Fig. 6
figure 6

Minor characters’ precision and recall

Full size image

To compare with other proposed approaches, while RoleNet [6] create major, supporting roles and identifies characters that belong to the corresponding groups. RoleNet produces a weighted graph where nodes represent the characters in the movie, the edges represent the relationships among the characters, and the weights are the number of characters’ who co-occur in certain frames. There is a limitation to this method due in that the image processing accuracy rate is not yet adequate, so it is not possible to reliably discover the main characters in the given movie. Character-net [5] consists of an approach that can be used to extract information from the social network by considering the dialogs among the characters in the movie. The dialog in the given movie contains the time, speaker and listener. These features could be extracted from the subtitles and the spoken lines of the characters through text mining and speech recognition. Character-net automatically classifies characters into several roles, including major roles, minor roles and extra roles. Character-net is a directed graph that has nodes representing characters and edges representing relationships among the characters. The weights and directions illustrate the direction of the dialog in the movie. This method only considers dialogs in which the characters have action/co-occurrence. However, in certain kinds of movies (i.e., horror, action, thriller), there is a lack of sufficient dialog among the characters. In contrast to RoleNet and Character-net, our method is more reliable because we annotate all of the appearances and disappearances of the characters during the movie. By doing this, we can consider all of the times during which characters appear. Thus, we can provide a better analysis of the movie content and relationships among the characters. Based on these results, the main story of the movie can be discovered.

Results of extracting summarization version of the given movie are illustrated in Table 2 and Fig. 7. For comparing to other approaches, the scene-based and RoleNet method [6], where the authors have provided methods to use face recognition to detect the appearance of the characters in the movie and to classify characters who have leading roles. However, the researchers were not able to discover the main character (or protagonist) and the main storyline of the movie. Other methods have provided similar results using scripts or face recognition to discover co-occurrences of characters in a movie, but these methods also have some limitations as above [5]. On the other hand, our approach can be used for truncating by an average of 75% to provide the summarized version, as illustrated in Fig. 7 and Table 2. The results of summarization of this work are better in comparing to those that have been previously obtained [23], with a 65% summarization level.

Table 2 Movie summarization
Full size table

In addition, we designed the user survey questions for measuring user satisfaction as the following: “How did the understanding of the given movie by watching the summarization?”. For measuring this question, we asked 27 users to get the feedback by ranking how did they understand the given movie (by scoring in range of [1...5]). Results of evaluation experimental showed that our study should extract the summarization version while the users still understand the given movie. Figure 8 illustrates the satisfaction of the user to our proposed method.

Fig. 7
figure 7

Movie summarization

Full size image
Fig. 8
figure 8

User survey evaluation

Full size image

4.2 Limitation and extension

Movie indexing and summarization is challenging work because it is not easy to produce an efficient result of the summarization. Our proposed method is based on character annotation and is not the only strategy that can be used to discover character relationships and the appearances of characters “on-screen” visually. Other techniques that should be adopted include script mining, subtitle analysis, image processing or speech recognition. These methods can be used to discover whether a character appears in a scene based on text mining, face recognition, and speaker identified, as was previously performed in [5] and [6]. The movie script, subtitle analysis and annotations can be combined with the character appearance to discover and construct the characters network so that a more reasonable and efficient method can be developed in the future.

Other problems of the proposed method are that it does not consider the character’s emotions and behaviors/activities. The proposed method can be combined with subtitle and script analysis in the future to more effectively discover information from the movie.

5 Conclusion

In our study, we provided a work that analyzes and discovers the main storyline of the given movie based on the characters’ appearances and the co-appearances by analyzing the character network. In this regard, centralities of the social network and the sub-plots were discovered to find the most important characters in the movie, and these results were used to mine the main storyline of the movie. The movie summarization and indexing follow the main storyline, which revolves around the protagonist. The novel features of our research that are different from that in existing study is that others have focused on user preferences to evaluate the results. The results of other researches depend on the behavior of various users and on user activity (i.e., [10] and [15]). As a result, the authors do not consider the main storyline of the movie and are unable to discover the main characters. Besides, we use the user survey question to measure the understanding of user for each summarization version. However, our study has several limitations because it takes time to annotate the movies other features of characters in the movie are not considered including behavior or emotion. These issues will be solved in the future study.