Advertisement

Remembering winter was coming

Character-oriented video summaries of TV series
  • Xavier BostEmail author
  • Serigne Gueye
  • Vincent Labatut
  • Martha Larson
  • Georges Linarès
  • Damien Malinas
  • Raphaël Roth
Article

Abstract

Today’s popular tv series tend to develop continuous, complex plots spanning several seasons, but are often viewed in controlled and discontinuous conditions. Consequently, most viewers need to be re-immersed in the story before watching a new season. Although discussions with friends and family can help, we observe that most viewers make extensive use of summaries to re-engage with the plot. Automatic generation of video summaries of tv series’ complex stories requires, first, modeling the dynamics of the plot and, second, extracting relevant sequences. In this paper, we tackle plot modeling by considering the social network of interactions between the characters involved in the narrative: substantial, durable changes in a major character’s social environment suggest a new development relevant for the summary. Once identified, these major stages in each character’s storyline can be used as a basis for completing the summary with related sequences. Our algorithm combines such social network analysis with filmmaking grammar to automatically generate character-oriented video summaries of tv series from partially annotated data. We carry out evaluation with a user study in a real-world scenario: a large sample of viewers were asked to rank video summaries centered on five characters of the popular tv series Game of Thrones, a few weeks before the new, sixth season was released. Our results reveal the ability of character-oriented summaries to re-engage viewers in television series and confirm the contributions of modeling the plot content and exploiting stylistic patterns to identify salient sequences.

Keywords

Extractive summarization tv series Plot analysis Dynamic social network 

Notes

Acknowledgements

This work was supported by the Research Federation Agorantic, Avignon University.

References

  1. 1.
    Balas E, Zemel E (1980) An algorithm for large zero-one knapsack problems. Oper Res 28(5):1130–1154.  https://doi.org/10.1287/opre.28.5.1130 MathSciNetzbMATHGoogle Scholar
  2. 2.
    Bost X (2016) A storytelling machine ? : automatic video summarization : the case of TV series. Theses, Université d’Avignon . https://tel.archives-ouvertes.fr/tel-01637270
  3. 3.
    Bost X, Labatut V, Gueye S, Linarès G (2018) Extraction and analysis of dynamic conversational networks from tv series. In: Kaya M, Kawash J, Khoury S, Day MY (eds) Social network based big data analysis and applications. Springer International Publishing, Cham, pp 55–84.  https://doi.org/10.1007/978-3-319-78196-9_3
  4. 4.
    Bost X, Labatut V, Gueye S, Linarès G (2016) Narrative smoothing: dynamic conversational network for the analysis of tv series plots. In: 2016 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), pp 1111–1118.  https://doi.org/10.1109/ASONAM.2016.7752379
  5. 5.
    Bredin H, Gelly G (2016) Improving speaker diarization of tv series using talking-face detection and clustering. In: Proceedings of the 24th ACM international conference on multimedia, MM ’16. ACM, New York, pp 157–161,  https://doi.org/10.1145/2964284.2967202
  6. 6.
    Chen HW, Kuo JH, Chu WT, Wu JL (2004) Action movies segmentation and summarization based on tempo analysis. In: Proceedings of the 6th ACM SIGMM international workshop on multimedia information retrieval, MIR ’04. ACM, New York, pp 251–258,  https://doi.org/10.1145/1026711.1026752
  7. 7.
    Daskin MS (2011) Network and discrete location: models, algorithms, and applications. Wiley, New YorkzbMATHGoogle Scholar
  8. 8.
    Ercolessi P, Sénac C, Bredin H (2012) Toward plot de-interlacing in tv series using scenes clustering. In: 2012 10th international workshop on content-based multimedia indexing (CBMI), pp 1–6,  https://doi.org/10.1109/CBMI.2012.6269836
  9. 9.
    Fayard D, Plateau G (1982) An algorithm for the solution of the 0–1 knapsack problem. Computing 28(3):269–287.  https://doi.org/10.1007/BF02241754 zbMATHGoogle Scholar
  10. 10.
    Felzenszwalb P, McAllester D, Ramanan D (2008) A discriminatively trained, multiscale, deformable part model. In: 2008 IEEE conference on computer vision and pattern recognition, pp 1–8.  https://doi.org/10.1109/CVPR.2008.4587597
  11. 11.
    Friedland G, Gottlieb L, Janin A (2009) Using artistic markers and speaker identification for narrative-theme navigation of seinfeld episodes. In: 2009 11th IEEE international symposium on multimedia, pp 511–516,  https://doi.org/10.1109/ISM.2009.20
  12. 12.
    Giannakopoulos T, Pikrakis A, Theodoridis S (2008) Music tracking in audio streams from movies. In: 2008 IEEE 10th workshop on multimedia signal processing, pp 950–955.  https://doi.org/10.1109/MMSP.2008.4665211
  13. 13.
    Gillick D, Favre B (2009) A scalable global model for summarization. In: Proceedings of the workshop on integer linear programming for natural langauge processing, ILP ’09. Association for Computational Linguistics, Stroudsburg, pp 10–18,  https://doi.org/10.3115/1611638.1611640. http://dl.acm.org/citation.cfm?id=1611638.1611640
  14. 14.
    Guha T, Kumar N, Narayanan SS, Smith SL (2015) Computationally deconstructing movie narratives: an informatics approach. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 2264–2268,  https://doi.org/10.1109/ICASSP.2015.7178374
  15. 15.
    Hanjalic A (2005) Affective video content representation and modeling. IEEE Trans Multimedia 7(1):143–154.  https://doi.org/10.1109/TMM.2004.840618 Google Scholar
  16. 16.
    Hanjalic A, Lagendijk RL, Biemond J (1999) Automated high-level movie segmentation for advanced video-retrieval systems. IEEE Trans Circuits Syst Video Technol 9(4):580–588.  https://doi.org/10.1109/76.767124 Google Scholar
  17. 17.
    Koprinska I, Carrato S (2001) Temporal video segmentation: a survey. Signal Process Image Commun 16(5):477–500.  https://doi.org/10.1016/S0923-5965(00)00011-4. http://www.sciencedirect.com/science/article/pii/S0923596500000114 Google Scholar
  18. 18.
    Lartillot O, Toiviainen P, Eerola T (2008) A matlab toolbox for music information retrieval. In: Preisach C, Burkhardt H, Schmidt-Thieme L, Decker R (eds) Data analysis, machine learning and applications. Springer, Berlin, pp 261–268,  https://doi.org/10.1007/978-3-540-78246-9_31
  19. 19.
    Ma YF, Lu L, Zhang HJ, Li M (2002) A user attention model for video summarization. In: Proceedings of the 10th ACM international conference on multimedia, MM ’02. ACM, New York, pp 533–542,  https://doi.org/10.1145/641007.641116
  20. 20.
    McDonald R (2007) A study of global inference algorithms in multi-document summarization. In: Amati G, Carpineto C, Romano G (eds) Advances in information retrieval. Springer, Berlin, pp 557–564,  https://doi.org/10.1007/978-3-540-71496-5_51
  21. 21.
    Sang J, Xu C (2010) Character-based movie summarization. In: Proceedings of the 18th ACM international conference on multimedia, MM ’10.  https://doi.org/10.1145/1873951.1874096. ACM, New York, pp 855–858
  22. 22.
    Smeaton AF, Lehane B, O’Connor NE, Brady C, Craig G (2006) Automatically selecting shots for action movie trailers. In: Proceedings of the 8th ACM international workshop on multimedia information retrieval, MIR ’06. ACM, New York, pp 231–238,  https://doi.org/10.1145/1178677.1178709
  23. 23.
    Smith JR, Joshi D, Huet B, Hsu W, Cota J (2017) Harnessing a.i. for augmenting creativity: Application to movie trailer creation. In: Proceedings of the 25th ACM international conference on multimedia, MM ’17. ACM, New York, pp 1799–1808,  https://doi.org/10.1145/3123266.3127906
  24. 24.
    Tapaswi M, Bäuml M, Stiefelhagen R (2014) Storygraphs: visualizing character interactions as a timeline. In: 2014 IEEE conference on computer vision and pattern recognition, pp 827–834.  https://doi.org/10.1109/CVPR.2014.111
  25. 25.
    Tsoneva T, Barbieri M, Weda H (2007) Automated summarization of narrative video on a semantic level. In: International conference on semantic computing (ICSC 2007), pp 169–176.  https://doi.org/10.1109/ICSC.2007.42
  26. 26.
    Weng C, Chu W, Wu J (2009) Rolenet: movie analysis from the perspective of social networks. IEEE Trans Multimed 11(2):256–271.  https://doi.org/10.1109/TMM.2008.2009684 Google Scholar
  27. 27.
    Yeung M, Yeo BL, Liu B (1998) Segmentation of video by clustering and graph analysis. Comput Vis Image Underst 71(1):94–109.  https://doi.org/10.1006/cviu.1997.0628. http://www.sciencedirect.com/science/article/pii/S1077314297906287 Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.OrkisAix-en-ProvenceFrance
  2. 2.Laboratoire Informatique d’AvignonAvignon UniversityAvignonFrance
  3. 3.Intelligent Systems DepartmentDelft University of TechnologyDelftThe Netherlands
  4. 4.Centre for Language StudiesRadboud University NijmegenNijmegenThe Netherlands
  5. 5.Institute for Computing and Information SciencesRadboud University NijmegenNijmegenThe Netherlands
  6. 6.Centre Norbert EliasAvignon UniversityAvignonFrance

Personalised recommendations